Software Testing

What are Load, Stress, Volume, etc. testing?

Pavankumar Pothuraju — Wed, 16 Nov 2005 20:14:00 GMT

- Volume testing is a way to test functionality.

- Stress testing is a way to test reliability.

- Load testing is a way to test performance.

Testing an application under heavy but expected loads is known as load
testing. It generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the system's services concurrently. As such, load testing is most relevant for a multi-user system, often one built using a client/server model, such as a web server tested under a range of loads to determine at what point the system's response time degrades or fails. Although you could perform a load test on a word processor by or graphics editor forcing it read in an extremely large document; on a financial package by forcing to generate a report based on several years' worth of data, etc.

When the load placed on the system is accelerated beyond normal usage
patterns, in order to test the system's response at unusually high or peak loads, it is known as stress testing. Stress testing is often incorrectly used interchangeably with load and performance testing.

In a stress test, the load is usually so great that error conditions are the expected result, although there is a grey area between the two domains and no clear boundary exists where you could say that an activity ceases to be a load test and becomes a stress test.

There is little agreement on what the specific goals of load testing are. The term is often used synonymously with performance testing, reliability testing, and volume testing.

Performance testing is usually performed to determine how fast some aspect of a system performs under a particular workload. It can serve different purposes: to demonstrate that the system meets performance criteria; to compare two systems to find which performs better; to measure what parts of the system or workload cause the system to perform badly.

In the diagnostic case, software engineers use tools such as profilers to measure what parts of a device or software contribute most to the poor performance. In performance testing, it is often crucial (and often difficult to arrange) for the test conditions to be similar to the expected actual use.

A reliability test determines how likely a piece of hardware (or sometimes software) is to fail. It is part of reliability theory, used originally as a tool to help nineteenth century insurance companies compute profitable rates to charge their customers. Statistical models appropriate for any of these are generically called "time-to-event" models. Death or failure is called an "event", and the goal is to project or forecast the rate of events for a given population or probability of an event for an individual.

Volume testing is used to determine if the system under test can handle the required amounts of data, user requests, etc.

Soak testing occurs when running a system at normal to high levels of load for prolonged periods of time. A soak test would normally execute several times more transactions in an entire day than would be expected in a busy day. This should identify any performance problems that appear after a large number of transactions have been executed.

For some of the basics on this type of testing, please read the following series:

Web Page Response Time 101
http://www.stickyminds.com/r.asp?F=DART_5030

Web Load Test Planning
http://www.stickyminds.com/r.asp?F=DART_5084

The Science and Art of Web Site Load Testing
http://www.stickyminds.com/r.asp?F=DART_1939

Getting Started : Stressing Web Applications Stress Early -- Stress Often
http://www.amibug.com/getinfo.shtm?present=webstress

http://www.perftestplus.com
http://www.performancetester.com

Testing Client/Server Systems

Pavankumar Pothuraju — Thu, 21 Apr 2005 02:35:00 GMT

Courtesy: Paul Gerrard, Systeme Evolutif Ltd. www.evolutif.co.uk

What's different about testing client/server?

Client/server architectures allow complex systems to be assembled from components. However, multiple operating systems, changing technologies and greater architectural complexity make integration more difficult. Risks such as poor reliability, performance, configuration management, security and other non-functional issues are more prominent. None of the risks in client/server are new, but there has been a change in emphasis. Since its purpose is to address risk, the emphasis of testing in client/server must change.

The complexity of client/server also makes testing more difficult. Complex systems can be delivered faster because they are assembled from bought-in components. Testing is often estimated to be in proportion with the development cost but in a typical client/server system, the development cost may be small compared with the overall cost of the system. The end-result in such projects, is that testers are presented with more complex systems, but the time left for testing is squeezed. In client/server, testers often have to do MORE WITH LESS!

Client/server test process

A client/server test strategy must identify the risks of concern and define a test process that ensures these risks are addressed. It is axiomatic that a problem is cheaper to fix if identified early, so the test process should be aligned very closely to the development process. Testing of a deliverable should occur as soon as possible after it has been built.

Functionality is usually tested in three stages: a unit or component test, often performed by programmers; a system test, of the complete system in a controlled environment performed by a dedicated test resource; finally, an acceptance test under close to production conditions, often performed by users. The objectives, techniques and responsibility for these three test stages are directly comparable to test stages in more traditional host-based systems.

The changed emphasis in testing client/server is associated with integration and non-functional testing.

Integration is a big issue because client/server systems are usually assembled from around twelve components (for a simple 2-Tier system) to perhaps twenty components for a complex architecture. These components are usually sourced from multiple suppliers. Although standards are emerging, client/server architectures often use components which have never been used before in combination.

The sheer number of interfaces involved make interface problems and inter-component conflicts more likely. Assumptions made by developers of one component from one supplier may contradict assumptions made by another developer. These problems may be encountered for the first time ever in your installation. Getting suppliers to take such problems seriously may be difficult because in client/server, it is the system integrator who takes responsibility for integration testing.

Performance consistently presents a problem in client/server. `Intelligent clients' may call for and process large volumes of data across networks. The amount of code exercised in large numbers of architectural layers may be substantial. Delays between distributed processes may be only ten or twenty milliseconds, but when a transaction requires hundreds of network messages, delays can be considerable.

Other non-functional issues such as security, backup and recovery and system administration also present risks. What was taken for granted in a mainframe environment often presents problems in client/server.

The test strategy must address all these risks, but testing does not happen `at the end'. Testing occurs at all stages and includes reviews, walkthroughs and inspections. Developers should be responsible for the products they deliver and should test their own code. System tests should cover non-functional areas as well as the functionality. The overarching principle is test early, test often.

Managing testing

To ensure testing gets attention, define testing as a specific activity within the development phase. Instil a regime that says components are delivered and acceptable only when tested without error and signed off.

Promote better test practices within the development team. Developers are often reluctant (or incapable) of testing their own code, so implement a `buddy scheme', where pairs of programmers test each others code. If practical, put testers into the development team to obtain the benefits of independence.

When a project moves from the development phase into system and acceptance testing, the nature of the project changes. Incident logging, management and reporting become the project manager's main method of project control. Progress is measured by tests performed, errors detected, fixed and retested. A simple PC database is usually adequate for recording incidents and providing useful progress reports and statistics.

Without a strong configuration management and change control regime, systematic testing can be ineffective. Complexity makes this more important, but also more difficult. Ensure a definitive inventory of components exists, system builds are automated where possible and releases are base-levelled.

Test automation

If you are not planning to conduct formal, documented functional tests, do not buy a tool to `do regression testing'. It will probably be a distraction, and lose you both time and money. If you do intend to implement regression tests, formulate a highly selective and economically viable regression test policy and only attempt to create automated regression tests when the software is sufficiently stable.

Integration and interface testing using test drivers or harnesses should be encouraged. Given test running and dynamic analysis tools, your developers can usually build test harnesses quickly. Where object-oriented languages, such as C++ are used, dynamic analysis tools are essential to detect memory leaks and give programmers invaluable insights into the behaviour of their code. Consider developing extended or `soak' tests of components where bespoke, low level code has been written.

Architectural components should be load and stress tested individually and as a complete system if resilience or performance are of concern. Whether you adopt a top-down, bottom-up or other integration strategy, the principle is to gain confidence in one build configuration, before integrating the next components for test.

System performance testing requires considerable tool support. Load generation, response time measurement, server, database and client monitoring and performance analysis may all be required. Although several vendors offer sophisticated test running tools with load regulation and performance analysis facilities, there is no one solution available. A combination of proprietary test tools, in-house utilities and innovation are usually needed to build, execute and analyse comprehensive tests.

Performance testing and tuning tends to follow three stages. Early tests tend to cause failures in the system caused by bugs or gross mis-setting of system parameters. In stage two, application design errors, such as inefficient SQL are identified. Finally, when the system is reliable and the application tuned, the database and server operating system parameters can then be tuned to achieve optimal performance.

Testing skills

Testing is a skill which requires a disciplined, systematic approach. Few developers, testers or users have ever received formal training in testing. Training helps testers not only to do better testing but helps them to be more motivated. Users, in particular, can benefit from testing training because they will better understand how to organise themselves, to prepare more comprehensive tests and be more critical of the systems they are expected to accept. If RAD is used, responsibility for testing shifts from the developer to the user, so user testing skills are essential.

Professional Tester - Magazine for testing professionals

Pavankumar Pothuraju — Tue, 29 Mar 2005 21:01:00 GMT

Professional Tester is a magazine for testing professionals. Anyone can subscribe to this magazine and receive by email. Articles from the January 2004 are available for DOWNLOAD

Some good links on performance testing (Load testing)

Pavankumar Pothuraju — Tue, 29 Mar 2005 20:29:00 GMT

1) http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vsent7/html/vxcontestingforperformance.asp

2) http://msdn.microsoft.com/msdnmag/issues/03/01/LoadTesting/

3) http://www.softwaretesting.nildram.co.uk/cont383.htm (Good site, refer for other testing concepts too)

4) http://www.theserverside.com/news/thread.tss?thread_id=3396 (Performance Testing with the MS Web Application Stress Tool)

5) http://www.theserverside.com/news/thread.tss?thread_id=23988 (Performance Testing your Distributed J2EE Applications)

6) http://www.theserverside.com/articles/article.tss?l=Tips-On-Performance-Testing-And-Optimization (Tips on Performance Testing and Optimization)

Interview Questions: Testing and QA

Pavankumar Pothuraju — Fri, 10 Dec 2004 17:30:00 GMT

QA & Testing related questions

http://www.techinterviews.com/index.php?p=120

Test Automation Interview Questions

http://www.techinterviews.com/index.php?p=116

Load Runner Interview Questions

http://www.techinterviews.com/index.php?p=118

You can also find questions on categries like .Net, C++, Database, General, Hardware, Java, Networking, Testing, Unix/Linux, VB, and Windows. For more details refer www.techinterviews.com

Testing E-Commerce Systems: A Practical Guide

Pavankumar Pothuraju — Fri, 03 Dec 2004 20:59:00 GMT

by Wing Lam

As e-customers (whether business or consumer), we are unlikely to have confidence in a Web site that suffers frequent downtime, hangs in the middle of a transaction, or has a poor sense of usability. Testing, therefore, has a crucial role in the overall development process. Given unlimited time and resources, you could test a system to exhaustion. However, most projects operate within fixed budgets and time scales, so project managers need a systematic and cost-effective approach to testing that maximizes test confidence.

This article provides a quick and practical introduction to testing medium- to large-scale transactional e-commerce systems based on project experiences developing tailored solutions for B2C Web retailing and B2B procurement. Typical of most e-commerce systems, the application architecture includes front-end content delivery and management systems, and back-end transaction processing and legacy integration.

Aimed primarily at project and test managers, this article explains how to

establish a systematic test process, and
test e-commerce systems.

The Test Process

You would normally expect to spend between 25 to 40 percent of total project effort on testing and validation activities. Most seasoned project managers would agree that test planning needs to be carried out early in the project lifecycle. This will ensure the time needed for test preparation (establishing a test environment, finding test personnel, writing test scripts, and so on) before any testing can actually start.

The different kinds of testing (unit, system, functional, black-box, and others) are well documented in the literature. However, less well articulated (despite the good general texts in software engineering) is the test process that ties together all testing and validation activities used to deliver a quality solution to the client.

Table 1 presents a matrix model of the test process that my colleagues and I have come to adopt on several e-commerce projects. Several goals (listed in the table’s first column) guide this model; one or more processes support the achievement of each goal.

Table 1. Matrix model of the test process
Goal	Processes
Set and agree on realistic expectations for the system	Prioritize business requirements; identify those that are essential and those that are optional. For a given delivery date, filter out, reprioritize, or renegotiate unrealistic system expectations. Specify requirements that are testable. Remove or firm up ambiguous requirements. Define acceptance criteria that are meaningful to the business. Define relevant industry standards or competitive benchmarks. Ensure that stakeholders agree to a capacity plan for the system with measurable capacity requirements.
Define a test strategy	Formulate and document a test strategy; relate it to the test quality plan. Identify high-risk zones early on; mark these for early testing and keep them away from the project's critical path. Identify all aspects of the system that require testing; in each case, indicate the level and form of testing. Identify relevant testing tools that can help automate or document elements of the test process.
Plan the testing	Estimate required resources, such as tools, people, hardware and software infrastructure, test location, and so on. Define roles and responsibilities; identify a suitably qualified test manager and test team. Draw up and agree to a detailed test plan with the client; define test milestones and test deliverables. Plan a pre-release simulation period where the system can be observed under simulated business usage.
Set up the test environment	Set up the physical test environment—for example, the hardware and software infrastructure. Prepare test material such as detailed test plans, test scripts, and test data. Prepare test materials with reuse in mind. Set up the defect-tracking system; decide on the form of defect reporting and any test metrics to use. Define test standards and procedures; educate individuals on the use of standards and procedures. Acquire and prepare the necessary test tools; ensure that the test team receives relevant training.
Perform testing	Conduct appropriate quality reviews, such as code reviews and walk-throughs. Conduct testing, whether at the unit, integration, system, or use-acceptance level. Execute tests and collate test results; record results and incidents in the defect-tracking system. Analyze test results; monitor areas with a high incidence of defects and defect severity. Track and manage defect resolution. Where appropriate, repeat the test and reuse test materials to expedite retesting. Stage a pre-live simulation period; test the system under the expected business usage and observe results. Manage the decision to go live based on testing results; consider if the system has achieved a release standard.
Monitor the deployed system	Monitor system performance using relevant monitoring tools; use performance reporting to guide performance-tuning activities. Capture any residual defects; assess their priority and, if serious, consider a rectification plan. Gather feedback and problem reports from users; analyze these to identify future improvements.
Manage successive releases	Capture and prioritize new business requirements; assess their impact on the system before committing to them. Fix outstanding and residual defects. Plan the system's next release; agree to a release plan with relevant stakeholders. Ensure a smooth transition from the current release to the new release.

We prefer the term “goals” to “phases” to indicate the fact that goals are concurrent. That is, you can be testing the system’s current release, but at the same time, setting and agreeing to realistic expectations for the next release.

The model—or at least parts of it—should be familiar to those experienced in test management roles. Those who aren’t or don’t follow a systematic test process can use the matrix model as a template for planning testing and validation activities by allocating processes to a project plan. It is also useful as a tool for improving a test process by checking the model against the actual process.

Testing an E-Commerce System

So how do you test an e-commerce system? What types of testing do you need to do? Here, I present a checklist of test areas that group together common types of testing normally applied to e-commerce systems. (Some of these tests also apply to other types of systems.)

You can cover certain test areas (such as browser compatibility) relatively quickly, whereas others (like those for transaction integrity) require a higher level of effort.

Browser compatibility

Customers will become frustrated if they can’t get to your e-commerce site easily. Though relatively simple to do, it pays to spend enough time testing in this area.

Lack of support for early browsers. Not all browsers are equal. For example, some early versions of Internet Explorer and Netscape Navigator do not fully support JavaScript. Decide on the lowest level of compatibility—for example, “the system should be compatible with IE version 3.0 and above and Netscape version 2.0 and above”—and test that the system does indeed work without problem on the early browser versions.

Browser-specific extensions. HTML is an evolving standard (HTML 4.0 is the latest specification) and should not be confused with HTML extensions specific to a particular browser implementation. For example, the table border color tag (<TABLE BORDER COLOR=“$$$$$$”>) is specific to Microsoft Internet Explorer; the Server script tag (<SERVER>) is an extension introduced with Navigator version 4.0.

Browser nuances. Even with the same release version, browsers behave differently on different platforms, and when used with different language options. Testing should cover at least the main platforms (Unix, Windows, Mac, and Linux) and the expected language options.

Page display

The display of pages in a browser forms the all-important interface between the customer and the business. Get this wrong in any way, and the job of building customer confidence and trust becomes much more difficult.

Incorrect display of pages. Displaying all pages correctly is not as easy as it sounds—most e-commerce systems generate Web pages dynamically (that is, they access a database and display the results in a browser) rather than display static HTML. Problems can occur—for example, when the browser retrieves an empty result set from the database or when a database connection is unavailable.

A page display might also involve loading resources (such as applets) beforehand, and you could encounter problems when such resources are unavailable. Test that your system handles such exceptions appropriately.

Runtime error messages. Users get frustrated when a browser throws up unexpected error messages. Such error messages are generally unfriendly and not meaningful to users; moreover, they leave a poor impression on the customer. Carry out tests to ensure that the application captures and handles all errors by, for example, generating an appropriate and user-friendly error page.

Poor page download times. US Web tracker Zona Research estimates that page load times of 8 seconds or more combined with ISP download times could cause up to 30 percent of customers to leave a site before buying anything. Pages that have a high graphical content and/or use applets, though aesthetically pleasing, are often download problematic. Test download time under realistic test conditions (that is, account for typical Internet traffic) rather than testing it locally.

Dead hyperlinks. Dead hyperlinks are a frequent cause of customer frustration; even the most popular portals suffer from hyperlinks that lead nowhere. Several automated tools now test hyperlinks.

Plug-in dependency. Developers gear some sites toward browsers that have a particular plug-in—the Flash graphics plug-in or language plug-ins are some examples. However, it is unreasonable to turn away a potential e-customer simply because the site is unusable without the plug-in. Test that a site is functionally equivalent with and without the plug-in.

Aesthetics. Sites need to be aesthetically pleasing—those that aren’t run the risk of conveying a poor image or, even worse, damaging an existing brand image. Where two competing sites are otherwise equal, a user’s choice may rest on aesthetic appeal alone. Though somewhat subjective, aesthetic reviews against competitive sites are valuable and force you to look at what competitors are doing.

Font sizing. Most browsers allow users to change font sizes. For example, in MS Internet Explorer, users can vary font sizes from “smallest” to “largest.” Designers often work with a particular font setting on their browser; consequently, site designs can often break-up in different font settings.

Session management

HTTP is a stateless protocol, and server-side programming tools that use Java or Active Server Pages make extensive use of session objects to capture state information. This is, for example, how many pages store items in a shopping cart. Our experience indicates that any system using session management should be tested for several characteristics.

Session expiration. Most applications and Web servers configure sessions so that they expire after a set time. Attempting to access a session object that has expired causes an error, and must be handled within the code; often it’s not. Developers tend to overlook testing for session expiration, largely because under normal operational circumstances, session expiration is unlikely to occur.

Session storage. Consider if any issues relate to the storing of session objects (which often depend on the application software). For example, session objects may be stored as cookies, but what if users do not have cookies enabled in their browsers? Also, are there any limitations to the physical size of session objects?

Usability

Good site usability is crucial for attracting and retaining customers; indeed, one study indicated that 67 percent of customers did not complete online purchases because of poor usability.

Usability covers a broad area and is often hard to define—or never defined at all. However, this lack of definition conflicts with testing and the need to have testable requirements. (If you can’t test requirements, how will you know that they have been met?)

As a rule of thumb, usability is the ease with which a customer can perform a set of tasks, typically determined by the time taken for successful completion. You can test usability through usability studies, in which you give a group of individuals selected tasks to carry out, observe their performance, and collect feedback. Alternatively, you can obtain this information in joint facilitation sessions in which Web site designers sit down with users immediately after a usability session and obtain direct verbal feedback on the site’s usability.

Nonintuitive design. You can label aspects of a Web site as nonintuitive if, on average, it takes a new visitor a disproportionate amount of time to perform set tasks. When designers become closely involved in a site’s design, it becomes harder for them to objectively assess the site’s intuitiveness.

Poor site navigation. Site navigation is crucial for attracting customers and retaining them. Sophisticated Web sites, such as for travel booking, need to pay particular attention to navigation issues. (See the “Tips for Better Site Navigation” sidebar.)

Catalog navigation. Large product catalogs are central to many e-commerce systems, B2B procurement systems in particular. Customers should be able to quickly browse and search through catalogs. Developers can define tests to measure the effectiveness of product navigation mechanisms. For example, you could test that a search on particular keywords brings up the correct products.

Lack of help. It’s not always obvious what to do on a Web site. For example, users may need to fill out forms in a particular sequence because they have associated validation rules that raise errors when some information is not correctly filled in. Guiding a customer through interactions is a key factor for first-time visitors.

Content analysis

So far, I’ve been concerned with the delivery and presentation of content. However, the actual content provided by a site also needs to be “tested.”

Offensive, misleading, and litigious content. An e-commerce site’s contents must be sound. In the UK, legislation such as the Trade Marks Action (1994), Control of Misleading Advertisements Regulations (1988), and Obscene Publications Act (1976) applies to all published material, including Web content. (See current UK- and EU-specific legislation.) Products or services that are incorrectly described or misleading (check those product descriptions!) may violate the Trade Description Act.

Infringements. Test your Web site content for infringements. Graphics and images used in the design should be royalty free (if not wholly owned); also check for copyright infringement. In some circumstances, the courts will consider use of trademarked material without the owner’s written consent as infringement. Courts would almost certainly view discrediting or denigrating a competitor’s brand as an infringement.

Personalization. Increasingly, e-commerce systems are incorporating customer relationship management functionality that provides personalization for individual customers. In many cases, a personalization profile determines what content the site offers to these customers. However, situations can occur where the site offers incorrect or inappropriate content to customers because of errors in the personalization functionality.

Availability

Unavailability equals lost revenue; it also harms a business’s reputation and can encourage customers to take their business to competitors. Businesses need to offer 24/7 availability to their customers, with availability levels of 99 percent or higher being the desired norm.

Unacceptable levels of unavailability. Several factors can influence availability, such as hardware reliability, software reliability, the effectiveness of load balancing, and the database’s ability to handle concurrent users. Before going live, predicted business usage patterns should indicate maximum stress levels. You should test system availability against the maximum stress levels plus a safety margin for a defined period of time.

Denial of service. Yahoo, Amazon.com, and Buy.com have all recently suffered denial-of-service attacks. Such attacks (also known as saturation attacks) involve bombarding the Web server with bogus requests, making it inaccessible to genuine users. Organizations are not defenseless against such attacks; Amazon.com for example, uses security software to filter out bogus requests. Test your e-commerce systems for vulnerability to denial-of-service attacks.

Backup and recovery

You can’t guarantee that any component of your e-commerce system won’t fail, whether hardware or software, so it’s sensible to test that you can quickly recover from a failure when it does happen.

Failure/fall-over recovery. Design systems so that one component’s failure doesn’t necessarily bring down other components. In MS Internet Information Server (version 4.0), for example, Active Server Page scripts can run from the Web server as separate processes.

Test both the speed and ease at which systems can recover from component failure; for example, how do you restart the service? Is it a case of pressing the restart button, or should the system support team look at other factors first? Is it a simple administrative task, or one that requires specialist knowledge? For non-fatal failures, you might be able to reload the system’s affected components rather than taking the system down completely. For example, in a Java implementation, you can reload Java servlets dynamically.

Backup failure. How quick and easy it is to perform and restore backups? If database records become corrupted, for example, how quickly can you backup the data? How quickly can new hardware, software, and networking components replace failed components?

Fault tolerance. In fault-tolerant architectures, redundant hardware and software components take over when components fail. For example, if one Web server goes down, a second Web server can service requests, and the problem remains transparent to the user. Test fault tolerance by switching off individual components or simulating their failure. My colleagues and I know of one incident where the system used clustering technology to provide fault tolerance. However, testing found that the clustering technology was ineffective because it was incorrectly configured.

Transactions

Transaction processing is a central element of most e-commerce systems. With distributed systems, transaction processing can span several individual systems, leading to more complex testing.

Transaction integrity. Does the system correctly perform all aspects of transaction processing? For example, does the system call the appropriate database triggers and procedures? Does it commit and roll back transactions correctly? Have system developers set the appropriate isolation levels? Devise a set of inputs that will attempt to create “nonstandard” transactions—for example, transactions with a zero quantity or value—as well as standard ones.

Throughput. The most popular Internet sites easily exceed a million hits per day. In a transactional e-commerce system, you must test the transaction engine’s throughput; that is, its ability to manage, say, 1,000 transactions per minute. Most modern testing and monitoring tools have facilities for throughput testing.

Auditing. An audit trail records all transactions made by the system. Tests should verify that the audits satisfy all legal obligations, and that the auditing software captures all the required transaction details, such as transaction time and date, originator and recipient, values, and so on.

Shopping, order processing, and purchasing

My experience suggests that functional testing typically consumes between 25 and 50 percent of the total testing effort. In most e-commerce systems, shopping, order processing, and purchasing form the core functionality. Other functional areas can include customer profiling, discount and offer management, automatic marketing, and inventory management. Although most developers largely consider functional testing a manual process, tools (such as Mercury Winrunner) can often help automate aspects of functional testing by automatically capturing and replaying user interactions.

Shopping-cart functionality. Many e-commerce sites use shopping carts (or some variant along the same theme) as a mechanism for the customer to compile orders. Basic shopping carts provide “list and total-up” functionality, whereas more sophisticated ones can provide extra functionality, such as tax and shipping calculation, loyalty discounts, and “save and recall” features for orders. Testing should cover all aspects of shopping-cart functionality based on test suites of shopping items.

Order processing. Order processing can involve the automated creation or update of transactions, or the delivery of information to back-end systems.

Payment processing. Approaches to payment processing vary from the simple capture of credit card details for offline manual processing to real-time card processing. Irrespective of the chosen approach, you must test several basic actions, such as determining whether the system carries out the necessary validation and debits the correct amount from the credit card. The log must also correctly record each transaction.

Order tracking. An increasingly common feature in B2B systems is the ability to track orders. One point of testing is the accuracy of the order-tracking functionality. You need to enter test orders into the system, and compare the automated system’s status report to the actual order status.

Internationalization

E-commerce systems, especially B2C systems, are often required to operate across geographic boundaries. Although English is the dominant language, non-English sites are increasingly common and gaining a head start in cornering foreign markets. A system must operate in selected target countries as well as it does in its home country.

Language support. Does the site offer the option to view non-English pages? If the choice of language is based on browser preferences, does it work on all the desired browsers? Many older browsers do not support language customization.

Language display. Test that words are correctly displayed and that sentences are grammatically correct. Use a native speaker to verify that this is the case.

Cultural sensitivity. Does the site respect cultural differences between the home and foreign country? Check that a site does not include content that is likely to offend the culture or religious affiliation of the country concerned.

Regional accounting. Developing e-commerce systems where trade takes place across accounting regions raises issues about how to charge customers. For example, tests should ensure that the system correctly calculates taxes and exchange rates for the regions concerned.

Operational business procedures

E-commerce business procedures often involve both manual and automated procedures. A telephone help desk, for example, can handle product returns or order inquires, but relies on information from the automated processes. Don’t forget that you must test e-business procedures, not just the e-commerce system per se.

As a first step, identify the various business scenarios that can take place. For example, a customer can request a product return (over the phone, or via e-mail or other communications means) or cancel an order. Payments can be delayed or canceled.

Next, try and estimate the size and scale of each scenario: How many returns are you likely to get during a peak period? Simulate the peak situation and see how well the e-business procedure copes. In particular, observe where the main bottlenecks are and where you can make the most value-added improvements. Before going live, ensure that you know at least the rough capacity of operational business procedures.

Systems integration

Developers must often integrate e-commerce systems with existing back-office or legacy systems, such as accounting and stock-control systems. Or, you could need to integrate e-commerce systems with third-party applications, such as e-mail. In some cases, such as in cross-platform integration, you can employ specialized middleware technology such as CORBA (the Common Object Request Broker Architecture) or messaging (implemented in software such as IBM MQ Series) to integrate systems.

Data interface format. The data interface defines the format of data exchanged by front- and back-end systems. In a B2B scenario, for example, we must ensure that an order between a supplier and a consumer conforms to an agreed order format. Such interfaces can vary from simple to highly complex, and tests should ensure that the system generates and transmits data in the correct format. Tools such as XML (Extensible Mark-Up Language) alleviate data interface problems by providing document type definitions (DTDs).

Interface frequency and activation. The processing between front- and back-end systems may be time dependent. For example, a back-end system could be necessary to process a data transmission from the front-end system immediately or within a defined period. Tests should ascertain whether a system actually observes timeliness constraints and whether it activates data transmissions at the correct time.

Updates. One system must often update information in another system. Verify that batch programs and remote procedures perform the necessary update operations without side effects.

Interface volume capacity. Do the interfaces handle the volume of data envisaged? Bear in mind that legacy back-end systems may not have been designed for the volume of data typical today.

Integrated performance. The performance of two systems integrated together can be a key issue. For example, you may need to integrate an e-commerce system with the payment systems of a financial processor to process credit cards online.

In another scenario, employing middleware technology such as CORBA can incur unacceptable overhead by using object brokers. Perform tests under simulated business usage to ensure acceptable integration performance.

Performance

The performance that matters the most is the end performance delivered to the customer, and Internet traffic makes this performance variable. Testing’s goal is to ascertain a system’s performance degradation profile (how fast and how much it degrades) and system breakpoints, which are the thresholds over which a system component, hardware or software, will crash or stop functioning.

Performance bottlenecks. An e-commerce system is only as strong as its weakest link. For example, performance suffers when you pair a Web server that can handle thousands of multiple users with a database that can handle only 20 concurrent connections. Effective testing involves identifying and stressing the weakest links to avoid overspending on the strong links.

Load handling. A large UK catalog firm’s supply chain became unable to cope with order processing in time for Christmas 1999. This situation forced the business to turn away customers in mid-December. Determining the rate of performance degradation (the degradation profile) and a system’s breakpoints is essential to achieving specific quality of service (QoS) requirements. For example, you can specify a QoS requirement such as, “Performance will drop no more than 10 percent when the number of concurrent users reaches 10,000.”

You can use tools such as Apache JMeter and WebLoad to test server performance both on static and dynamic resources (such as CGI scripts, servlets, or Perl scripts) under different load types. You can ramp up the traffic on a Web site from 100 to 1,000 simultaneous users, depending on your estimates for normal and peak business usage.

Scalability analysis. E-commerce systems that do not scale well are limited from day one. Scalability analysis involves “testing” a system’s architectural design and its capability to meet future (rather than current) performance requirements. Basic areas of scalability are hardware, application software, databases, and middleware.

Horizontal scalability involves adding new components, such as servers. Vertical scalability concerns how existing components can be grown, for example, by adding more disks to a disk array. Scalability analysis seeks to identify where—and how easily—you can scale the architecture.

Login and security

Security (or a lack of it) is a major barrier to e-commerce, particularly in Europe. With the rise in credit card fraud and high-profile hacker attacks, customers increasingly avoid e-commerce sites and systems they perceive as insecure.

Login/registration capability. Several potential problems can arise from failed login functionality. The most typical of such problems is a user providing a valid login and password but not being able to log in. Other problems can be more complex, such as when the system sets personalization settings incorrectly. Where appropriate, tests should also verify that the system correctly sets privilege levels.

Penetration and access control. Security personnel define penetration as unauthorized access to restricted areas and information. External penetration refers to attacks from the outside (over the Internet); internal penetration refers to attacks from within.

A large recruitment agency recently damaged its image when hackers gained unauthorized access (over the Internet and without password) to its resume database. This database contained sensitive information on workers’ personal details and salary.

We also know of a case where an intruder gained access to unauthorized areas by simply typing in a URL directly rather than going through the intended route via the home page. You should test security measures against unauthorized access to determine the actual level of protection.

Insecure information transmission. Transfer of information occurs from browser to server, server to browser, server to back-end system and vice versa. For some, Secure Sockets Layer (SSL) has become the de facto standard for secure online communications. Where information is sensitive, you must test against stipulated security requirements. What level of encryption is required, and does the encryption system meet these requirements?

Web attacks. How prone is the system to Web attacks such as spoofing, snooping, and sniffing? Where Web attacks are a high priority (such as for financial systems), consider a security evaluation from an independent security organization. Some security tools, for example, check for sniffers and highlight insecure network configurations.

Computer viruses. How resilient is the system against computer viruses? Test that any virus protection software loads at the appropriate time, and that the necessary procedures (manual or automated) to update virus files are in place.

Digital signatures. The use of digital signatures is on the rise, particularly in secure document exchange applications where authentication and nonrepudiation are key. Tests should cover the means by which digital signatures are generated and issued. They should also check whether certificate files are loaded and installed correctly, and that the application handles invalid signatures. Don’t assume that it will work; test that it does!

The production of high-quality e-commerce systems relies on comprehensive and well-managed testing. I strongly suggest that project and test managers treat the ideas presented here not as a definitive guide, but as a framework for fleshing out, rethinking, or evaluating their own test models and strategies. We have a simple rule of thumb about testing: If in doubt, you need to do more testing.

The risk-based testing

Pavankumar Pothuraju — Fri, 15 Oct 2004 20:34:00 GMT

1. Make a prioritized list of risks.

2. Perform testing that explores each risk.

3. As risks evaporate and new ones emerge, adjust your test effort to stay focused on the current crop.

Any questions? Well, now that you know what risk-based testing is, I can devote the rest of the article to explaining why you might want to do it, and how to do it well.

Why Do Risk-Based Testing?

As a tester, there are certain things you must do. Those things vary depending on the kind of project you’re on, your industry niche, and so on. But no matter what else you do, your job includes finding important problems in the product. Risk is a problem that might happen. The magnitude of a risk is a joint function of the likelihood and impact of the problem—the more likely the problem is to happen, and the more impact it will have if it happens, the higher the risk associated with that problem. Thus, testing is motivated by risk. If you accept this premise, you might well wonder how the term "risk-based testing" is not merely redundant. Isn’t all testing risk-based?

To answer that, look at food. We all have to eat to live. But it would seem odd to say that we do "food-based living." Under normal circumstances, we don’t think of ourselves as living from meal to meal. Many of us don’t keep records of the food we eat, or carefully associate our food with our daily activities. However, when we are prone to eat too much, or we suffer food allergies, or when we are in danger of running out of food, then we may well plan our lives explicitly around our next meal. It is the same with risk and testing.

Just because testing is motivated by risk does not mean that explicit accounting of risks is required in order to organize a test process. Standard approaches to testing are implicitly designed to address risks. You may manage those risks just fine by organizing the tests around functions, requirements, structural components, or even a set of predefined tests that never change. This is especially true if the risks you face are already well understood or the total risk is not too high.

If you want higher confidence that you are testing the right things at the right time, risk-based testing can help. It focuses and justifies test effort in terms of the mission of testing itself. Use it when other methods of organizing your effort demand more time or resources than you can afford.

If you are responsible for testing a product where the impact of failure is extremely high, you might want to use a rigorous form of risk analysis. Such methods apply statistical models and/or comprehensively analyze hazards and failure modes. I’ve never been on a project where we felt the cost of rigorous analysis was justified, so all I know about it is what I’ve read. One well-written and accessible book on this subject is Safety-Critical Computer Systems by Neil Storey.

There is also a technique of statistically justified testing taught by John Musa in his book Software Reliability Engineering.

There is another sort of risk analysis about which relatively little has been written. This kind of analysis is always available to you, no calculator required. I call it heuristic risk analysis.

Heuristic Analysis

A heuristic method for finding a solution is a useful method that doesn’t always work. This term goes back to Greek philosophers, but George Polya introduced it into modern usage in his classic work How to Solve It. Polya writes, "Heuristic reasoning is reasoning not regarded as final and strict but as provisional and plausible only, whose purpose is to discover the solution of the present problem."

Heuristics are often presented as a checklist of open-ended questions, suggestions, or guidewords. A heuristic checklist is not the same as a checklist of actions that you might include as "steps to reproduce" in a bug report. Its purpose is not to control your actions, but to help you consider more possibilities and interesting aspects of the problem. For a wonderful set of heuristics for developing software requirements, see Exploring Requirements: Quality Before Design, by Don Gause and Gerald M. Weinberg.

Two Approaches to Analysis

Let’s look at some heuristics for exploring software risk. I think of risk analysis as either "inside-out" or "outside-in." These are complementary approaches, each with its own strengths.

Inside-Out
Begin with details about the situation and identify risks associated with them. With this approach, you study a product and repeatedly ask yourself "What can go wrong here?" More specifically, for each part of the product, ask these three questions:

Vulnerabilities What weaknesses or possible failures are there in this component?
Threats What inputs or situations could there be that might exploit a vulnerability and trigger a failure in this component?
Victims Who or what would be impacted by potential failures and how bad would that be?

This approach requires substantial technical insight, but not necessarily your insight. The times I’ve been most successful with inside-out risk analysis were when making "stone soup" with a developer. I brought the stones (the heuristics); he brought the soup (the facts).

Here’s what that looks like: In a typical analysis session we find an empty conference room that has a big whiteboard. I ask "How does this feature work?" The developer then draws a lot of scrunched boxes, wavy arrows, crooked cylinders, and other semi-legible symbology on the board. As he draws, he narrates the internal workings of the product.

Meanwhile, I try to simulate the mechanism in my head as fast as the developer describes it. When I think I understand the process or understand how to test it, I explain it back to him. The whiteboard is an important prop because I get confused easily as I’m assimilating all the information. When I lose the thread of the explanation, I can scowl mysteriously, point to any random part of the diagram, and say something like "I’m still not clear on how this part works."

As I come to understand the mechanism, I look for potential vulnerabilities, threats, and victims. More precisely, I make the developer look for them with questions such as:

[pointing at a box] What if the function in this box fails?
Can this function ever be invoked at the wrong time?
[pointing at any part of the diagram] What error checking do you do here?
[pointing at an arrow] What exactly does this arrow mean? What would happen if it were broken?
[pointing at a data flow] If the data going from here to there were somehow corrupted, how would you know? What would happen?
What’s the biggest load this process can handle?
What external components, services, states, or configurations does this process depend upon?
Can any of the resources or components diagrammed here be tampered with or influenced by any other process?
Is this a complete picture? What have you left out?
How do you test this as you’re putting it together?
What are you most worried about? What do you think I should test?

This is not a complete list of questions, but it’s a good start. Meanwhile, as the developer talks, I listen for whether he is operating on faith or on facts. I listen for any uncertainty or concern in his voice, hesitations, or a choice of words that may indicate that he has not thought through the whole problem of requirements, design, or implementation. Confusion or ambiguity suggests potential risk. When we identify a risk, we also talk about how I might test so as to evaluate and manage that risk.

A session like this lasts about an hour, usually—and I leave with an understanding of the feature, as well as a list of specific risks and associated test strategies. The tests I perform as a result of that conversation serve not only to focus on the risks, but also to refute or corroborate the developer’s story about the product.

There are wonderful advantages to this approach, but it requires effective communication skills on the part of the developer and tester, and a willingness to cooperate with each other. You can perform this analysis without the developer, but then you have the whole burden of studying, modeling, and analyzing the system by yourself.

Inside-out is a direct form of risk analysis. It asks "What risks are associated with this thing?" Inside-out is the opposite of the outside-in approach, which asks "What things are associated with this kind of risk?"

Outside-In
Begin with a set of potential risks and match them to the details of the situation. This is a more general approach than inside-out, and somewhat easier. With this approach, you consult a predefined list of risks and determine whether they apply here and now. The predefined list may be written down, or it may be something burned into your head by the flames of past experience. I use three kinds of lists: quality criteria categories, generic risk lists, and risk catalogs.

Quality Criteria Categories These categories are designed to evoke different kinds of requirements. What would happen if the requirements associated with any of these categories were not met? How much effort is justified in testing to assure they are met to a "good enough" standard?

Capability Can it perform the required functions?
Reliability Will it work well and resist failure in all required situations?
Usability How easy is it for a real user to use the product?
Performance How speedy and responsive is it?
Installability How easily can it be installed onto its target platform?
Compatibility How well does it work with external components and configurations?
Supportability How economical will it be to provide support to users of the product?
Testability How effectively can the product be tested?
Maintainability How economical will it be to build, fix, or enhance the product?
Portability How economical will it be to port or reuse the technology elsewhere?
Localizability How economical will it be to publish the product in another language?

I cobbled together this list from various sources including the ISO 9126 standard, Hewlett Packard’s FURPS list (Functionality, Usability, Reliability, Performance, Supportability), and a few other sources. There is nothing authoritative about it except that it includes all the areas I’ve found useful in desktop application testing. I remember this list using the acronym CRUPIC STeMPL. To memorize it, say the acronym out loud and imagine that it’s the name of a Romanian hockey player. With a little practice, you’ll be able to recall the list any time you need it.

Generic Risk Lists Generic risks are risks that are universal to any system. These are my favorite generic risks:
Complex - anything disproportionately large, intricate, or convoluted
New - anything that has no history in the product
Changed - anything that has been tampered with or "improved"
Upstream Dependency - anything whose failure will cause cascading failure in the rest of the system
Downstream Dependency - anything that is especially sensitive to failures in the rest of the system
Critical - anything whose failure could cause substantial damage
Precise - anything that must meet its requirements exactly
Popular - anything that will be used a lot
Strategic - anything that has special importance to your business, such as a feature that sets you apart from the competition
Third-party - anything used in the product, but developed outside the project
Distributed - anything spread out in time or space, yet whose elements must work together
Buggy - anything known to have a lot of problems
Recent Failure - anything with a recent history of failure

Risk Catalogs A risk catalog is an outline of risks that belong to a particular domain. Each line item in a risk catalog is the end of a sentence that begins with "We may experience the problem that..." Risk catalogs are motivated by testing the same technology pattern over and over again. You can put together a risk catalog just by categorizing the kinds of problems you have observed during testing. Here’s an example of part of an installation risk catalog:

(For an example of a very broad risk catalog, see Appendix A of Testing Computer Software by Cem Kaner, Jack Falk, and Hung Nguyen.)

Wrong files installed

Temporary files not cleaned up

Old files not cleaned up after upgrade

Unneeded file installed

Needed file not installed

Correct file installed in the wrong place

Files clobbered

Older file replaces newer file

User data file clobbered during upgrade

Other apps clobbered

File shared with another product is modified

File belonging to another product is deleted

Hardware not properly configured

Hardware clobbered for other apps

Hardware not set for installed app

Screen saver disrupts install
No detection of incompatible apps

Apps currently executing

Apps currently installed

Installer silently replaces or modifies critical files or parameters
Install process is too slow
Install process requires constant user monitoring
Install process is confusing

User interface is unorthodox

User interface is easily misused

Messages and instructions are confusing

You can use these risk lists in a number of ways. Here’s one that works for me:

1. Decide what component or function you want to analyze. Are you looking at the whole product, a single component, or a list of components?

2. Determine your scale of concern. I like to use a scale of "normal," "higher," and "lower." Everything is presumed to be a normal risk unless I have reason to believe it’s a higher or a lower risk. Use a scale that’s meaningful to you, but beware of ambiguous scales, or scales that appear more objective than they really are.

3. Gather information (or people with information) about the thing you want to analyze. Obviously, you need to know something about the situation in order to analyze it. When I’m doing "outside-in" analysis on a product, I gather whatever information is convenient, make a stab at the analysis, then go to the people who are more expert than I and have them critique the analysis. Another way to do this is to get all those people in the same room at the same time and do the analysis in that meeting.

4. Visit each risk area on each list and determine its importance in the situation at hand. For each area, ask "Could we have problems in this area? If so, how big is that risk?" Record your impression. Think of specific reasons that support your impression. If you’re doing this in a meeting, ask "How do we know that this is or is not a risk? What would we have to know in order to make a better risk estimate?"

5. If any other risks occur to you that aren’t on the lists, record them. Special risks are bound to occur to you during this process.

6. Record any unknowns, which impact your ability to analyze the risk. During the process, you will often feel stumped. For example, you might wonder whether a particular component is especially complex. Maybe it’s not complex at all. What do you need to know in order to determine that? As you go through the analysis, it helps to make a list of information-gathering "to do" items. At some point, go get that information and update your analysis.

7. Double-check the risk distribution. It’s common to end up with a list of risks in which everything is considered to be equally risky. That may indeed be the case. On the other hand, it may be that your distribution of concerns is skewed because you’re not willing to make tough choices about what to test and what not to test. Whatever distribution of risks you end up with, double-check it by taking a few examples of equal risks and asking whether those risks really are equal. Take some examples of risks that differ in magnitude and ask if it really does make sense to spend more time testing the higher risk and less time testing the lower risk. Confirm that the distribution of risk magnitudes feels right.

I recommend including a variety of people from a variety of roles in this analysis. Use people from Technical Support, Development, and Marketing, for instance.

Three Ways to Organize Risk-Based Testing

Whether you employ outside-in, inside-out, or some hybrid approach to doing the analysis, I can suggest three different ways to communicate the risks and organize the testing around those risks: risk watch list, risk/task matrix, or component risk matrix.

Risk Watch List
This is probably the simplest way to organize risk-based testing. A risk watch list is just a list of risks that you periodically review to ask yourself what your testing has revealed about those issues. If you feel you don’t have enough recent information about problems in the product that are associated with a risk, then do some more testing to gather that information.

Risk/Task Matrix
The risk/task matrix consists of a table with two columns. On the left is a list of risks; on the right is a list of risk mitigation tasks associated with each risk. Sort the risks by importance, with the most important risks at the top. Think of each row in the matrix as a statement of the form "If we’re worried about risk X, then we should invest in tasks Y."

The risk/task matrix is useful mainly as a tool in negotiating for testing resources. I like using this technique in situations where Management would not accept poor testing, yet also would not provide enough testing staff to do that job. The matrix helps bring management expectations in line with available resources. It’s a lot easier to get testing resources when you can explain the impact of not having enough.

A disadvantage of this approach is that some tasks mitigate more than one risk. Also, some mitigation tasks cost so much or take so much time that they actually add more problems to the project than they’re worth in terms of the problems they help detect. Still, it’s a simple way to show the gross relationships between risk and test effort on a project-wide basis.

Component Risk Matrix
The component risk matrix consists of a table with three columns. Break the product into thirty or forty areas or components. These components can be physical code (such as "the install program"), functions (such as "print"), or data (such as "clipart library"). In other words, a component is anything that is subject to testing. In the leftmost column of each row of the matrix, list a component. In the rightmost column, list all of the known risk heuristics that indicate significant risk in that component (if a risk heuristic applies equally to all components, don’t bother listing it). In the middle column, write a summary risk judgment of "higher," "lower," or "normal" (see Table 1).

Table 1

Component	Risk	Risk Heuristics
Printing	Normal	Distributed, Popular
Report Generation	Higher	New, Strategic, Third-party, Complex, Critical
Installation	Lower	Popular, Usability, Changed
ClipArtLibrary	Lower	Complex

What this matrix helps you do is help communicate and negotiate which components will get more effort. I use a general rule that higher-risk items get twice the effort as normal items, which in turn get twice the effort as the components that are lower-risk. (This is just an approximation, of course.)

The risk heuristics are included in the table because they help provoke questions about your risk judgments, but remember—there is no hard relationship between the heuristics and any particular judgment. You may find yourself in a situation where you will argue that one component is more risky than another, even though the first component has more heuristics driving its risk than the second. Risk analysis is a matter of evaluating factors that influence risk, not merely counting them.

As the project proceeds, you pay testing attention to different components in rough accord with their associated levels of risk. A disadvantage of this approach is that it focuses only on highlighting risks that increase the need to test, and not on those factors that decrease the need to test. You could add those risk-lowering factors into the matrix, of course, but I find that it makes the matrix too complicated.

Making It All Work
Always keep this in mind: your risk analysis is going to be incomplete and inaccurate to some degree, and it may be very wrong. All you really have at the beginning of a project are rumors of risks. As the project progresses, and you gain information about the product, you should adjust your test effort to match your best estimation of risk. Also, to deal with the risk of poor risk analysis, don’t let risk-based testing be the only kind of testing you do. Spend at least a quarter of your effort on approaches that are not risk-focused—such as field testing, code coverage testing, or functional coverage testing. This is called the principle of diverse half-measures: use a diversity of methods because no single heuristic always works.

Finally, if I were to choose two vital factors needed to make risk-based testing work, I would name experience and teamwork. Over a period of time, any product line or technology will reveal its pattern of characteristic problems (assuming that you pay attention to problems found in the field). Learn from that. And do whatever you can to invite different people with different points of view into the risk analysis process.

If there’s a magic to risk-based testing, it’s the magic of noticing the signs and clues, all around you, about where the problems lie. Some people do this without consciously thinking about it, and maybe that’s good enough. But when a problem slips by you because you couldn’t do perfectly exhaustive testing, you may be called upon to explain why you did what you did. Management may assume that you did a sloppy job, and they may not be impressed with the standard argument that all testing is incomplete. That’s when it’s nice to have that risk list or matrix. With risk-based testing, you can show Management that you strive to make the best use of the resources they invest. They’ll respect you for that.

I thank Cem Kaner and Brian Lawrence for their advice and criticism in the development of this article.

About the Author James Bach is the founder of Satisfice, Inc., a test training and consulting company. A pioneer in the emerging disciplines of Good enough quality and exploratory testing, James specializes in expert testing under chaotic conditions. He can be reached at james@satisfice.com.

Validation & Verification

Pavankumar Pothuraju — Fri, 01 Oct 2004 23:33:00 GMT

S no.	Verification	Validation
1.	Verification is a static testing procedure.	Validation is dynamic testing procedure.
2.	It involves verifying the requirements, detailed design documents, test plans, walkthroughs and inspections of various documents produced during the development and testing process.	Validation involves actual testing of the product as per the test plan (unit test, integration test, system test and acceptance test etc).
3.	It is a preventive procedure.	It is a corrective procedure.
4.	Are we building the product RIGHT?	Are we building the RIGHT product?
5.	It involves more then two to three persons and is a group activity.	It involves the testers and sometimes user.
6.	It is also called Human testing, since it involves finding the errors by persons participating in a review or walk through.	It is also called Computer testing, since errors are found out by testing the software on a computer.
7.	Verification occurs on Requirements, Design and code.	Validation occurs only on code and the executable application.
8.	Verification is made both in the Executable and Non Executable forms of a work product	Validation is done only on Executable forms of a work product.
9.	Verification finds errors early in the requirement & design phase and hence reduces the cost of errors.	Validation finds errors only during the testing stage and hence cost of errors reduced is less than Verification.
10.	An effective tool for verification tool is a Checklist.	Various manual and automated test tools are available for Validation.
11.	It requires cooperation and scheduling of meetings and discussions.	It is to check that the product satisfies the requirements and is accepted by the user.
12.	Verification tasks include: 1) Planning 2) Execution	Validation tasks include: 1) Planning 2) Testware Development 3) Test Execution 4) Testware Maintenance
13.	Verification activities include: 1) Requirements Verification 2) Functional design verification 3) Internal Design Verification 4) Code Verification	Validation activities include: 1) Unit testing 2) Usability testing 3) Function testing 4) System testing 5) Acceptance testing
14.	Verification deliverables (work products) are: 1) Verification test plan 2) Inspection report 3) Verification test report	Validation deliverables are: 1) Test plan 2) Test Design Specification 3) Test Case Specification 4) Test Procedure Specification 5) Test log 6) Test incident report

Load and Stress testing

Pavankumar Pothuraju — Tue, 21 Sep 2004 21:27:00 GMT

Boris Beizer says:

One of the most common, but unfortunate misuse of terminology is treating "load testing" and "stress testing" as synonymous. The consequence of this ignorant semantic abuse is usually that the system is neither properly "load tested" nor subjected to a meaningful stress test.

1. Stress testing is subjecting a system to an unreasonable load while denying it the resources (e.g., RAM, disc, mips, interrupts, etc.) needed to process that load. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not expected to process the overload without adequate resources, but to behave (e.g., fail) in a decent manner (e.g., not corrupting or losing data). Bugs and failure modes discovered under stress testing may or may not be repaired depending on the application, the failure mode, consequences, etc. The load (incoming transaction stream) in stress testing is often deliberately distorted so as to force the system into resource depletion.

2. Load testing is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean representative load," "overload," "high load," etc. In performance testing, load is varied from a minimum (zero) to the maximum level the system can sustain without running out of resources or having, transactions suffer (application-specific) excessive delay.

3. A third use of the term is as a test whose objective is to determine the maximum sustainable load the system can handle. In this usage, "load testing" is merely testing at the highest transaction arrival rate in performance testing.

Integration Testing

Pavankumar Pothuraju — Mon, 30 Aug 2004 18:40:00 GMT

Integration Testing is an important step in a project, validating that the boundaries of a system are functioning properly and will deliver the required results. Specifically, all components within the system should cooperate with each other properly. Integration Testing is performed by Integration Testers or Software Developers.

Integration testing is a logical extension of Unit Testing. In its simplest form, two units that have already been tested are combined into a Component and the interface between them is tested. A component, in this sense, refers to an integrated aggregate of more than one unit.

In a realistic scenario, many units are combined into components, which are in turn aggregated into even larger parts of the program. The idea is totest combinations of pieces and eventually expand the process to test your modules with those of other groups. Eventually all the modulesmaking up a process are tested together. Beyond that, if the program is composed of more than one process, they should be tested in pairs rather than all at once.

Integration testing identifies problems that occur when units are combined. By using a test plan that requires you to test each unit and ensure the viability of each before combining units, you can assume that any errors discovered when combining units are likely related to theinterface between units. This method reduces the number of places tolook for the problem.

You can do integration testing in a variety of ways but the following are three common strategies:

The top-down approach to integration testing requires the highest-level modules be test and integrated first. This allows high-level logic and data flow to be tested early in the process and it tends to minimize the need for drivers. However, the need for stubs complicates test management and low-level utilities are tested relatively late in the development cycle. Another disadvantage of top-down integration testing is its poor support for early release of limited functionality. STUB is must for this approach.

The bottom-up approach requires the lowest-level units be tested and integrated first. These units are frequently referred to as utility modules. By using this approach, utility modules are tested early in the development process and the need for stubs is minimized. The downside,however, is that the need for drivers complicates test management and high-level logic and data flow are tested late. Like the top-down approach, the bottom-up approach also provides poor support for early release of limited functionality. DRIVER is must for this approach.

The third approach, sometimes referred to as the umbrella approach, requires testing along functional data and control-flow paths. First, the inputs for functions are integrated in the bottom-up pattern discussed above. The outputs for each function are then integrated in the top-down manner. The primary advantage of this approach is the degree of support for early release of limited functionality. It also helps minimize the need for stubs and drivers. The potential weaknesses of this approach are significant, however, in that it can be less systematic than the other two approaches, leading to the need for more regression testing.

Thanks to praneet_srivastava@mcafee.com

Kumar Praneet Srivastav
Senior Software Engineer (QA)