Testing & Debugging

BOOK REVIEW: Don't Make Me Think, 2d Ed. by Steve Krug

Chris Falter — Wed, 14 May 2008 03:31:39 GMT

You've mastered web forms and controls. You've prototyped a Silverlight 2.0 application. AJAX? You're all over it. But have you really learned how to design a good web page or web site?

Steve Krug's "Common Sense Approach to Web Usability" provides surprising and sometimes counterintuitive principles that every good website must follow. Krug preaches the importance of removing clutter in order to make the purpose and functionality of a site (or page) clear--and happily, he practices what he preaches in this remarkably lucid book. Here are some of Krug's key insights:

Don't make users think! Your job is to make sure users do not have to puzzle over a site's purpose, or how to use it. Krug offers the search functionality at Amazon.com as a great example of this principle in action. A user does not have to decide what type of search she wants (by author, by title, by ISBN, etc.); instead, she can just enter whatever text interests her, and Amazon offers a list of matches, ranked by relevance.
Users don't behave the way you think they do. You've been poring over your site--reading everything ten times or more--so you tend to think that users will do the same. But they don't. Instead, the users...
- "don't read pages, they scan them." (In fact you only decided to read this review after you scanned the intro and decided it would be worth your while.)
- "don't figure out how things work, they muddle through."
Design pages for scanning. Since users are going to treat your site like a billboard going by at 70mph (rather than a textbook that they carefully work through), you should learn to design great billboards!
- Create a clear visual hierarchy. Highlight the important stuff, and indicate relationship by grouping.
- Use conventions. If your site design conforms to what users generally expect, they can more easily understand it at a glance.
- Break pages into clearly defined areas.
- Make what's clickable obvious. Use arrows or underlines to indicate that text is clickable, for example.
- Minimize noise. Prefer clarity to pizzazz.
Give users simple\mindless choices. It's okay to make a user traverse 4 or 5 links to get to his desired destination as long as each choice along the way is clear.
"Omit needless words." "Get rid of half the words on each page, then get rid of half of what's left" is Krug's Third Law of Usability. More words just make the site look more daunting, which can discourage the user in a hurry. Krug's Third Law has 2 corollaries:
- Happy talk must die. "We're so glad you're at our site! We think you'll like your experience here..." Oops, you just lost your user, who's too busy to keep reading.
- Instructions must die. Users almost never read them anyway (remember, they don't figure out, they muddle through). So keep instructions brief and simple.

Krug then discusses some details about 2 features every site must have: navigation and a home page. Krug recommends that the sections and subsections of a site be indicated with a clear set of links or tabs on the home page, and that the navigation remain available no matter where the user goes. (This answers the questions: what can I do on this site? and Where can I go from here?) Each page should have a visible name, and the site should a breadcrumb trail with arrows between the levels in order to allow a user to know where he's at in the site's hierarchy. (These answer the questions: where am I at now? and How do I get back to where I was before?).

Krug's discussion of the home page acknowledges that the forces conspiring against simplicity are enormous in any ambitious site, because there are so many organizational interests competing for the valuable home page real estate. Krug provides some useful tips for managing the problems, though, by suggesting how to use logos, taglines, and home page navigation.

Next Krug addresses the often religious arguments that site development teams often endure (pulldowns or menus? Flash animation or simple text?) with this simple advice: forget the arguments and start testing! Simple usability testing will reveal the key problems with a site, and often they have nothing to do with what the development team has been arguing about. Krug believes that frequent tests are more important than comprehensive (often expensive tests), and he offers advice on how to do testing on a tight budget (use 3 or 4 subjects; use an inexpensive screen recorder like Camtasia; try to get all the project stakeholders to observe). He concludes with an extremely useful script of an interaction between a test subject and a test guide.

Those who have already read the first edition will be pleased to know that Krug has included some very helpful new material in the second edition. Is your site accessible to sight-impaired users? If not, Krug offers a top 5 list of tips for making sites accessible, along with a pointer on how to use Cascading Style Sheets to make your site more accessible. Not to mention that CSS makes your site a lot easier to manage.... And what do you do when your boss (or the customer) wants you to do something that violates every known law of web usability? Just compose an email that borrows liberally from one of Krug's friendly "here's how it should be done" missives!

Krug sprinkles his book with examples of sites that work well (and a few that don't) to illustrate his ideas. He often offers a few "How does (or does not) this page implement the principles we've been discussing?" tests, with his own answers on the following pages. I found these examples to be enormously helpful. And Krug offers a wonderful set of additional resources for those who want to pursue the subject further, both within the text and in a notated bibliography at the end.

Because I am a long-time web user and web developer, I thought I understood just about everything I needed to know about usability to design a good site. Then I read this book, and learned a ton. If you work on web sites in any way (whether as designer, developer, or tester), the few hours you spend on Steve Krug's little gem will pay rich dividends.

Why You Should Consider Using FIT

Chris Falter — Wed, 06 Feb 2008 04:12:48 GMT

I was looking for ways to improve our development process when I stumbled across the Framework for Integrated Testing (FIT). The rest of this post is the body of an email I recently sent to my manager.

The goal of the Customer update is to migrate all their customizations to a new framework in a reliable and quick manner. The major risks I have been able to identify are as follows:

Developers won’t be able to identify all the behaviors that need to migrate.
- Requirements are scattered.
- Code is not organized optimally. Much business and workflow logic is scattered throughout the UI layer, in particular. Many customizations in domain logic are located in poor locations in the domain class hierarchy; the root domain object seems to have become a catch-all location for just about anything.
Manual regression testing will become a bottleneck. It is already a serious bottleneck for other projects.
Poor code organization will be perpetuated into the new source code base. As a result, the project will proceed more slowly (due to high bug rate and unnecessary complexity), and subsequent evolution of the code will be hampered as well.
Lack of an automated testing framework will impede any efforts to refactor code. Since this project by definition requires an extensive reworking of our source code, this risk is quite serious.

Note to readers: I'm sure many of you have faced these kinds of challenges on one or more projects. Read on to discover a framework that can help you can meet the challenges!

While unit testing would go a long way toward addressing the third and fourth risks, it would not help us address the first 2 risks. We can ask business analysts (BAs) to help developers identify migration requirements, but we don’t have a repository for storing these requirements. Even if we implement such a repository, a developer must still translate requirements (in whatever form they exist) into unit tests, which introduces the possibility of error. Finally, the unit tests are not visible to the BAs, so they cannot rely on them for regression testing.

The Framework for Integrated Testing (FIT) seems to address all four of these risks quite adeptly. Before I proceed, I should explain that FIT allows a non-developer to write requirements in tabular form. FIT provides 3 types of test fixtures that can be used in a table:

A ColumnFixture is useful for business rules, as it is used to declare a set of inputs and expected output(s). For example, you could use a column fixture to declare that a dwelling in a particular location built in 1996 should have a BCEGS of 99, but if it is built in 2004 it should have a BCEGS of 4. As the Customer systems are replete with custom rules (such as underwriter referral rules), the ColumnFixture should prove very helpful.
A RowFixture is ensures that data are being stored and retrieved correctly from a data repository. While I see no need to test the data layer’s reliability with respect to standard data such as dwelling address (it has proven quite reliable for other projects), the RowFixture could help us with data that are unique to Customer (such as subdivision).
An ActionFixture is used to simulate workflow requirements. For example, a tabular ActionFixture could state that when a user enters a dwelling limit of $10k and presses “Continue”, a certain error message should appear; when s/he enters a dwelling limit of 300k and presses “Continue”, the system advances to the Quote Summary page; and if s/he enters a dwelling limit of 1100k, the system advances to the Qualification Questions page (due to consent to rate rules).

If we use the WinFitRunnerLite tool, a BA can construct the test data tables in Excel, which may be more intuitive than writing them in Word and saving as HTML.

A developer must write a test fixture in order to translate the test data into an actual test. The test fixture will link to the domain, data, and controller/workflow libraries of the system under test, pass data from the test document’s tables to the appropriate methods, and evaluate the results against the BA-supplied expectations. If we follow this approach, it will be impossible for a developer to embed workflow logic, rules, or domain logic in the UI (a big win!).

One of the other advantages of FIT is that a BA can “own” the generation of requirements and tests, since they are maintained in a single location. A BA can easily design the test scenarios that the system must pass; I’ve seen complex requirements in Excel spreadsheets plenty of times, so I have no doubt about the ability of our BAs to generate FIT scenarios. On the other hand, how many BAs could write a unit test?

Using FIT can help answer many project management questions:

Developer: How have requirements for the system changed? Answer: check the FIT document.
Developer: What are the requirements I should work on? Answer: see which FIT tests have not yet passed.
BA: How close are we to completing the latest project milestone? Answer: run the FIT tests and see how many tests have not yet passed.
BA: Do the latest changes contain any regressions? Answer: run the FIT tests and see if any tests that used to pass have started to fail.

It is true that the FIT tests do not run against the actual UI. I see this is a positive, as it will prevent the project team from falling into the “Smart UI” trap. However, it also means that FIT, however useful I think it will be, is not a silver bullet; testers will still have to interact with the system’s UI.

Nevertheless, I think FIT will prove very helpful for this project, and for our shop in general. And since BAs have to write requirements and perform testing, and developers have to translate requirements into testable code anyway, using FIT should not be burdensome.

A Tool for Troubleshooting Web Apps and Web Services

Chris Falter — Mon, 28 Jan 2008 19:20:45 GMT

Recently I discovered a tool that I had needed for a long time. Fiddler, a freeware product created and supported by Microsoftie Eric Lawson, can be used to debug HTTP traffic from any web browser (or client application). Here's how you can use it:

1. View/Analyze HTTP traffic.

One of our web apps creates a new window and performs a javascript-driven auto-logon to a vendor application. When we reported an issue to the vendor, they started asking questions about whether we were using the correct version of their javascript module. Rather than attempting to debug the javascript, I simply used Fiddler to capture all the HTTP traffic between my browser and the vendor's web server. Fiddler formatted the data quite nicely, which made verifying my browser's behavior quite simple. Fiddler was also able to save the relevant portions of the HTTP conversation to a log file, which I forwarded to the vendor. The vendor was able to determine quickly that the error was in the behavior of their web server, rather than in any incorrect use of javascript on our part.

Just one day later, another vendor questioned whether one of our web services was correctly returning error data. Source code-level debugging in the web service would have done me no good, since I had to verify the data being transmitted on the wire. But Fiddler made the investigation easy. I used Fiddler to capture an interaction between a test client and the service, and quickly verified that our service was returning the correct error data.

You can also use Fiddler to view all the details of a secure HTTP interaction. Try doing that with netmon! Just configure Fiddler to view HTTPS traffic, and it will hook any secure HTTP transactions. It uses the "man-in-the-middle" strategy: it sets up a secure HTTP transaction between the browser and itself, then transmits the data it receives to the target URL over secure HTTP. Sitting in the middle, it is able to view all the data in the browser-server conversation. You just have to tell your browser to ignore the any certificate errors associated with the temporary certificate that Fiddler generates.

2. Generate test scripts.

The Visual Studio 2005 Test Edition has a tool that allow you to generate a test script for a web app by capturing the interaction between a browser and a web app. However, this tool only captures interactions (such as form submits) generated by clicking on links and buttons. So how do you capture javascript-driven AJAX interactions? Use Fiddler!

3. Analyze performance enhancements.

If you are trying to figure out how various browser settings will affect a website's performance, Fiddler can help. You can script Fiddler to intercept browser interactions and alter metadata, which allows you to perform a "what if" analysis on the fly. And Fiddler has a built-in timer to measure wall clock time during the interaction.

The Fiddler site has extensive documentation, including some good QuickStart videos. If you've been looking for a good tool to analyze HTTP traffic, look no more; go directly to the Fiddler download page and get started!

Test-Driven Design: Make Sure You Fail!

Chris Falter — Sun, 21 Oct 2007 01:39:53 GMT

The idea behind test-driven design (TDD) is to incorporate testing into the process of constructing your system, rather than waiting for developers to check in their code and to build a system that someone else tests. TDD has some advantages worth considering:

Defects are identified sooner in the development process, which makes delivery of a reliable system cheaper and quicker.
Developers have more scope to improve the quality of code by refactoring, since they can run unit tests as they are making changes to verify whether the system still behaves as expected.
Unit tests serve as built-in documentation on how to call and make use of interfaces.
This built-in documentation is guaranteed not to go out of date. (You've never had the joy of working with out-of-date documentation, have you? You haven't lived until you've debugged some code whose documentation does not match. Or perhaps I should say, you haven't worked in software development unless you've debugged, etc.)
A team can run unit tests in the final step of the build process, in order to verify that the new build has no obvious defects.

The test-driven design process goes like this:

Get the requirement from the customer.
Design a test suite that will verify that the requirement is satisfied.
Run the test suite and make sure it fails.
Write code and run the test suite until it succeeds.

When the test suite succeeds and there are no obvious ways to simplify the code, the developer performs his customary victory dance and checks in the code.

I have incorporated TDD into the development process for some web services that have multiple customers. If we did not have a more or less complete suite of unit tests, any modification would be hard to manage. Any codepath that any customer relies on could have a regression when any code is changed, right? So in the absence of TDD, any modification would require that several testers (Pat, Tammy, and Charles at our office) spend a fair amount of their precious time running regression tests. Consequently, I have created suites of unit tests to verify that no regressions have occurred, and have even given them names ("virtual Pat," "virtual Tammy," and "virtual Charles") . Virtual Pat and her virtual colleagues are working hard so that the real Pat and her colleagues can head home at 5pm.

Recently I received a requirement to modify a service method so that it would return extra data. After I revised the unit tests, I almost skipped over step #3 (make sure the test fails). After all, I was in a hurry, and why bother? Aren't my coding skills good enough to write a simple test?

Evidently not.

I ran the test suite and it did not fail. I rubbed my eyes in disbelief and ran the suite again; of course, it succeeded again. A quick glance at the tests showed that the test suite contained the extra data expected from the method, but I had failed to write the assertions that would compare the expected data with the actual data returned from the method. So if the actual data returned from the method was not correct, the test (and the developer) would never know it. Doh! <Sound of hand smacking forehead />

So I added the assertions, ran the test suite (Hurrah! It failed!), wrote the new code, ran the test suite again, and it succeeded. I checked in my code, and it's working as expected.

The moral of this tale is simple: in test-driven design, you have to fail before you can succeed.