Best Practices in Software Test Automation

Siva — Sat, 13 Aug 2005 09:05:00 GMT

Very good article on best practices in Test Automation. Must read for everyone who is into test automation

http://www.testfocus.co.za/Feature%20articles/july2005.htm

Record and Play for Mozilla / FireFox Browser

Siva — Sat, 13 Aug 2005 08:49:00 GMT

Source: QAnews.com

AdventNet QEngine is the first tool to offer a single tool for record and play of Web Pages for the following environments.

Mozilla 1.5,1.6 and 1.7.3 [Linux and Windows ]
IE 6.0 [Windows]
QEngine release 5.1.0 (to be released by May 2005 ) wil have support for FireFox [ Windows and Linux ]

WinRunner Vs QTP

Siva — Fri, 05 Nov 2004 07:26:00 GMT

Check out this article on difference between Win Runner and QTP of Mercury Interactive:
http://www.asi-test.com/WinRunner_vs_QTP.htm

ATD - Automated Test Designer

Siva — Tue, 07 Sep 2004 23:16:00 GMT

ATD is a unique tool for creating Test Cases based on Functional Requirements, which can be used in a collaborative environment by all project team members and can handle changes and requirements of any complexity. It uses an advanced and rigorous Neural Network Optimization algorithm and reduction methods, in order to generate the minimum number of test cases to certify 100% of the Requirements rules.

Test Cases can be selected to meet available resources, budget and functional coverage allowing project and QA managers to control risk and target quality according to test strategies.

Anyone can download standalone version of the tool and try it for 30 days

"Homebrew Test Automation"

Siva — Tue, 13 Apr 2004 10:09:00 GMT

If 'B' represents what you need in order to perform satisfactory test automation for your own particular context and 'A' is the set of tools that are available to you, then "Homebrew Test Automation" is what you may need to create on your own in order to bridge the gap between 'A' and 'B'.

"Homebrew Test Automation" may include adding extensions to or modifying existing tools. It may also include writing new tools and/or integrating exiting tools in novel ways.

Whether the tools are commercial or free should be irrelevant. What is relevant is how easily the tools lend themselves to the above activities, either through providing source code, a well thought out extensibility interface or some combination, and whether you can ultimately demonstrate a positive return on investment in time and materials.

--written by Mike Silverstein in Stickyminds Roundtables

TEST AUTOMATION FRAMEWORKS

Siva — Tue, 04 Nov 2003 13:53:00 GMT

An excellent ebook by "Carl Nagle"

Manual or Automated?

Siva — Tue, 28 Oct 2003 08:27:00 GMT

Summary:Automated test tools are powerful aids to improving the return on the testing investment when used wisely. Some tests inherently require an automated approach to be effective, but others must be manual. In addition, automated testing projects that fail are expensive and politically dangerous. How can we recognize whether to automate a test or run it manually, and how much money should we spend on a test?

When Test Automation Makes Sense

Let’s start with the tests that ideally are automated. These include:

Regression and confirmation. Rerunning a test against a new release to ensure that behavior remains unbroken—or to confirm that a bug fix did indeed fix the underlying problem—is a perfect fit for automated testing. The business case for test automation outlined in Software Test Automation by Mark Fewster and Dorothy Graham is built around this kind of testing.

Monkey (or random). Tests that fire large amounts or long sequences of data, transactions, or other inputs at a system in a random search for errors are easily and profitably automated

Load, volume, and capacity. Sometimes, systems must support tremendous loads. On one project, we had to test how the system would respond to 50,000 simultaneous users, which ruled out manual testing! Two Linux systems running custom load-generating programs filled the bill.

Performance and reliability. With the rise of Web-based systems, more and more automated testing is aimed at looking for slow or flaky behavior on Web systems.

Structural, especially API-based unit, component, and integration. Most structural testing involves harnesses of some sort, which brings you most of the way into automation. Again, the article I wrote with Greg Kubaczkowski, "Mission Made Possible" (STQE magazine, July/Aug. 2002), provides an example.

Other tests that are well-suited for automation exist, such as the static testing of complexity and code standards compliance that I mentioned in the previous article. In general, automated tests have higher upfront costs—tools, test development, environments, and so forth—and lower costs to repeat the test.

When to Focus on Manual Testing

High per-test or maintenance costs are one indicator that a test should be done manually. Another is the need for human judgment to assess the correctness of the result or extensive, ongoing human intervention to keep the test running. For these reasons, the following tests are a good fit for manual testing:

Installation, setup, operations, and maintenance. In many cases, these tests involve loading CD-ROMs and tapes, changing hardware, and other ongoing hand-holding by the tester.

Configuration and compatibility. Like operations and maintenance testing, these tests require reconfiguring systems and networks, installing software and hardware, and so forth, all requiring human intervention.

Error handling and recovery. Again, the need to force errors—by powering off a server, for example—means that people must stay engaged during test execution.

Localization. Only a human tester with appropriate skills can decide whether a translation makes no sense, is culturally offensive, or is otherwise inappropriate. (Currency, date, and time testing can be automated, but the need to rerun these tests for regression is limited.)

Usability. As with localization, human judgment is needed to check for problems with the facility, simplicity, and elegance of the user interface and workflows.

Documentation and help. Like usability and localization, checking documentation requires human judgment.

Wildcards

In some cases, tests can be done manually, be automated, or both.

Functional. Functionality testing can often be automated, and automated functional testing is often part of an effort to create a regression test suite or smoke test. However, it makes sense to get the testing process under control manually before trying to automate functional testing. In addition, you’ll want to keep some of the testing manual.

Use cases (user scenarios). By stringing together functional tests into workflows, you can create realistic user scenarios, whether manual or automated. The trick here is to avoid automation if many workflows involve human intervention.

User interface. Basic testing of the user interface can be automated, but beware of frequent or extensive changes to the user interface that can incur high maintenance costs for your automated suite.

Date and time handling. If the test system can reset the computer’s clocks automatically, then you can automate these tests.

Higher per-test costs and needs for human skills, judgment, and interaction push towards manual testing. A need to repeat tests many times or reduce the cycle time for test execution pushes towards automated testing.

Reasons to Be Careful with Automation

Automated testing is a huge investment, one of the biggest that organizations make in testing. Tool licenses can easily hit six or seven figures. Neophytes can’t use most of these tools—regardless of what any glossy test tool brochure says—so training, consulting, and expert contractors can cost more than the tools themselves. Then there’s maintenance of the test scripts, which generally is more difficult and time consuming than maintaining manual test cases.