Test Notes

MSDN - Test Center

Siva — Mon, 29 Oct 2007 06:00:32 GMT

Microsoft MSDN Launched "Test Center" : A community where software testers can share knowledge and learn from each other.

New Software Testing Group

Siva — Fri, 18 Mar 2005 03:41:00 GMT

I have recently created a new Software Testing group. Let us share our knowledge using this group. Join this group and post only Software Testing/Quality/Automated Testing related questions......

Link: http://groups-beta.google.com/group/SoftwareTesting

XP Testing Without XP: Taking Advantage of Agile Testing Practices

Siva — Tue, 28 Oct 2003 09:27:00 GMT

An excellent article written by Lisa Crispin "XP Testing Without XP: Taking Advantage of Agile Testing Practices"

Test Metrics

Siva — Wed, 01 Feb 2006 11:44:00 GMT

One of the excellent resource for Testing Metrics. You will find different software testing metrics, their definition, purpose, and how to calculate. Must read. Click the following link

http://www.stpmag.com/downloads/stp-0507_testmetrics.htm

Mercury Interactive plans training centres

Siva — Fri, 01 Apr 2005 03:19:00 GMT

MERCURY Interactive Corporation, the US-based provider of automated software-testing solutions, plans to start authorised training centres to be run by franchisees for software testing in India.

The company hopes to have one or two centres in Delhi, Mumbai, Chennai, Bangalore and Hyderabad initially, according to Mr T. Srinivasan, Managing Director, Mercury India.

for more read this

Best Practices in Software Test Automation

Siva — Sat, 13 Aug 2005 09:05:00 GMT

Very good article on best practices in Test Automation. Must read for everyone who is into test automation

http://www.testfocus.co.za/Feature%20articles/july2005.htm

Record and Play for Mozilla / FireFox Browser

Siva — Sat, 13 Aug 2005 08:49:00 GMT

Source: QAnews.com

AdventNet QEngine is the first tool to offer a single tool for record and play of Web Pages for the following environments.

Mozilla 1.5,1.6 and 1.7.3 [Linux and Windows ]
IE 6.0 [Windows]
QEngine release 5.1.0 (to be released by May 2005 ) wil have support for FireFox [ Windows and Linux ]

Black Box Software Testing

Siva — Wed, 23 Mar 2005 07:49:00 GMT

The best testing course by Cem Kaner & James Bach

Quality Guru's

Siva — Tue, 28 Oct 2003 08:12:00 GMT

The early Americans

W Edwards Deming introduced concepts of variation to the Japanese and also a systematic approach to problem solving, which later became known as the Deming or PDCA cycle. Later in the West he concentrated on management issues and produced his famous 14 Points. He remains active today and he has attempted a summary of his 60 years experience in his System of Profound Knowledge

Deming encouraged the Japanese to adopt a systematic approach to problem solving, which later became known as the Deming or PDCA (Plan, Do, Check, Action) cycle. He also pushed senior managers to become actively involved in their company's quality improvement programmes.

Deming produced his 14 Points for Management, in order to help people understand and implement the necessary transformation. Deming said that adoption of, and action on, the 14 points are a signal that management intend to stay in business. They apply to small or large organisations, and to service industries as well as to manufacturing. However the 14 points should not be seen as the whole of his philosophy, or as a recipe for improvement. They need careful discussion in the context of one's own organisation.

Before his death Deming appears to have attempted a summary of his 60 years' experience. This he called the System of Profound Knowledge. It describes four interrelated parts:

Appreciation for a system
This emphasises the need for managers to understand the relationships between functions and activities. Everyone should understand that the long term aim is for everybody to gain - employees, share holders, customers, suppliers, and the environment. Failure to accomplish the aim causes loss to everybody in the system.

Knowledge of statistical theory
This includes knowledge about variation, process capability, control charts, interactions and loss function. All these need to be understood to accomplish effective leadership, teamwork etc.

Theory of knowledge
All plans require prediction based on past experience. An example of success cannot be successfully copied unless the theory is understood.

Knowledge of psychology
It is necessary to understand human interactions. Differences between people must be used for optimisation by leaders. People have intrinsic motivation to succeed in many areas. Extrinsic motivators in employment may smother intrinsic motivation. These include pay rises and performance grading, although these are sometimes viewed as a way out for managers.

Joseph M Juran focused on Quality Control as an integral part of management control in his lectures to the Japanese in the early 1950s. He believes that Quality does not happen by accident, it must be planned, and that Quality Planning is part of the trilogy of planning, control and improvement. He warns that there are no shortcuts to quality.

There are many aspects to Juran's message on quality. Intrinsic is the belief that quality does not happen by accident, it must be planned. His recent book Juran on Planning for Quality is perhaps the definitive guide to Juran's current thoughts and his structured approach to company-wide quality planning. His earlier Quality Control Handbook was much more technical in nature.

Juran sees quality planning as part of the quality trilogy of quality planning, quality control and quality improvement. The key elements in implementing company-wide strategic quality planning are in turn seen as identifying customers and their needs; establishing optimal quality goals; creating measurements of quality; planning processes capable of meeting quality goals under operating conditions; and producing continuing results in improved market share, premium prices, and a reduction of error rates in the office and factory.

Juran's Quality Planning Road Map consists of the following steps:

Identify who are the customers.

Determine the needs of those customers.

Translate those needs into our language.

Develop a product that can respond to those needs.

Optimise the product features so as to meet our needs as well as customer needs.

Develop a process which is able to produce the product.

Optimise the process.

Prove that the process can produce the product under operating conditions.

Transfer the process to Operations.

Illustration of Quality Trilogy via a Control Chart

Juran concentrates not just on the end customer, but identifies other external and internal customers. This effects his concept of quality since one must also consider the 'fitness of use' of the interim product for the following internal customers. He illustrates this idea via the Quality Spiral.

His formula for results is:

Establish specific goals to be reached.

Establish plans for reaching the goals.

Assign clear responsibility for meeting the goals.

Base the rewards on results achieved.

Dr Juran warns that there are no shortcuts to quality and is sceptical of companies that rush into applying Quality Circles, since he doubts their effectiveness in the West. He believes that the majority of quality problems are the fault of poor management, rather than poor workmanship on the shop-floor. In general, he believes that management controllable defects account for over 80% of the total quality problems. Thus he claims that Philip Crosby's Zero Defects approach does not help, since it is mistakenly based on the idea that the bulk of quality problems arise because workers are careless and not properly motivated.

Armand V Feigenbaum is the originator of Total Quality Control. He sees quality control as a business method rather than technically, and believes that quality has become the single most important force leading to organisational success and growth.

Dr Armand V Feigenbaum is the originator of Total Quality Control. The first edition of his book Total Quality Control was completed whilst he was still a doctoral student at MIT.

In his book Quality Control: Principles, Practices and Administration, Feigenbaum strove to move away from the then primary concern with technical methods of quality control, to quality control as a business method. Thus he emphasised the administrative viewpoint and considered human relations as a basic issue in quality control activities. Individual methods, such as statistics or preventive maintenance, are seen as only segments of a comprehensive quality control programme.

Quality control itself is defined as:
'An effective system for co-ordinating the quality maintenance and quality improvement efforts of the various groups in an organisation so as to enable production at the most economical levels which allow for full customer satisfaction.'

He stresses that quality does not mean best but best for the customer use and selling price. The word control in quality control represents a management tool with 4 steps:

Setting quality standards

Appraising conformance to these standards

Acting when standards are exceeded

Planning for improvements in the standards.

Quality control is seen as entering into all phases of the industrial production process, from customer specification and sale through design, engineering and assembly, and ending with shipment of product to a customer who is happy with it. Effective control over the factors affecting product quality is regarded as requiring controls at all important stages of the production process. These controls or jobs of quality control can be classified as:

New-design control

Incoming material control

Product control

Special process studies.

Quality is seen as having become the single most important force leading to organisational success and company growth in national and international markets. Further, it is argued that:

Quality is in its essence a way of managing the organisation and that, like finance and marketing, quality has now become an essential element of modern management.

Thus a Total Quality System is defined as:

The agreed company-wide and plantwide operating work structure, documented in effective, integrated technical and managerial procedures, for guiding the co-ordinated actions of the people, the machines and the information of the company and plant in the best and most practical ways to assure customer quality satisfaction and economical costs of quality.

Operating quality costs are divided into:

Prevention costs including quality planning.

Appraisal costs including inspection.

Internal failure costs including scrap and rework.

External failure costs including warranty costs, complaints etc.

Reductions in operating quality costs result from setting up a total quality system for two reasons:

Lack of existing effective customer-orientated customer standards may mean current quality of products is not optimal given use

Expenditure on prevention costs can lead to a severalfold reduction in internal and external failure costs.

The new 40th Anniversary edition of Dr A V Feigenbaum's book, Total Quality Control, now further defines TQC for the 1990s in the form of ten crucial benchmarks for total quality success. These are that:

Quality is a company-wide process.

Quality is what the customer says it is.

Quality and cost are a sum, not a difference.

Quality requires both individual and team zealotry.

Quality is a way of managing.

Quality and innovation are mutually dependent.

Quality is an ethic.

Quality requires continuous improvement.

Quality is the most cost-effective, least capital-intensive route to productivity.

Quality is implemented with a total system connected with customers and suppliers.

These are the ten benchmarks for total quality in the 1990s. They make quality a way of totally focusing the company on the customer - whether it be the end user or the man or woman at the next work station or next desk. Most importantly, they provide the company with foundation points for successful implementation of its international quality leadership.

Manual or Automated?

Siva — Tue, 28 Oct 2003 08:27:00 GMT

Summary:Automated test tools are powerful aids to improving the return on the testing investment when used wisely. Some tests inherently require an automated approach to be effective, but others must be manual. In addition, automated testing projects that fail are expensive and politically dangerous. How can we recognize whether to automate a test or run it manually, and how much money should we spend on a test?

When Test Automation Makes Sense

Let’s start with the tests that ideally are automated. These include:

Regression and confirmation. Rerunning a test against a new release to ensure that behavior remains unbroken—or to confirm that a bug fix did indeed fix the underlying problem—is a perfect fit for automated testing. The business case for test automation outlined in Software Test Automation by Mark Fewster and Dorothy Graham is built around this kind of testing.

Monkey (or random). Tests that fire large amounts or long sequences of data, transactions, or other inputs at a system in a random search for errors are easily and profitably automated

Load, volume, and capacity. Sometimes, systems must support tremendous loads. On one project, we had to test how the system would respond to 50,000 simultaneous users, which ruled out manual testing! Two Linux systems running custom load-generating programs filled the bill.

Performance and reliability. With the rise of Web-based systems, more and more automated testing is aimed at looking for slow or flaky behavior on Web systems.

Structural, especially API-based unit, component, and integration. Most structural testing involves harnesses of some sort, which brings you most of the way into automation. Again, the article I wrote with Greg Kubaczkowski, "Mission Made Possible" (STQE magazine, July/Aug. 2002), provides an example.

Other tests that are well-suited for automation exist, such as the static testing of complexity and code standards compliance that I mentioned in the previous article. In general, automated tests have higher upfront costs—tools, test development, environments, and so forth—and lower costs to repeat the test.

When to Focus on Manual Testing

High per-test or maintenance costs are one indicator that a test should be done manually. Another is the need for human judgment to assess the correctness of the result or extensive, ongoing human intervention to keep the test running. For these reasons, the following tests are a good fit for manual testing:

Installation, setup, operations, and maintenance. In many cases, these tests involve loading CD-ROMs and tapes, changing hardware, and other ongoing hand-holding by the tester.

Configuration and compatibility. Like operations and maintenance testing, these tests require reconfiguring systems and networks, installing software and hardware, and so forth, all requiring human intervention.

Error handling and recovery. Again, the need to force errors—by powering off a server, for example—means that people must stay engaged during test execution.

Localization. Only a human tester with appropriate skills can decide whether a translation makes no sense, is culturally offensive, or is otherwise inappropriate. (Currency, date, and time testing can be automated, but the need to rerun these tests for regression is limited.)

Usability. As with localization, human judgment is needed to check for problems with the facility, simplicity, and elegance of the user interface and workflows.

Documentation and help. Like usability and localization, checking documentation requires human judgment.

Wildcards

In some cases, tests can be done manually, be automated, or both.

Functional. Functionality testing can often be automated, and automated functional testing is often part of an effort to create a regression test suite or smoke test. However, it makes sense to get the testing process under control manually before trying to automate functional testing. In addition, you’ll want to keep some of the testing manual.

Use cases (user scenarios). By stringing together functional tests into workflows, you can create realistic user scenarios, whether manual or automated. The trick here is to avoid automation if many workflows involve human intervention.

User interface. Basic testing of the user interface can be automated, but beware of frequent or extensive changes to the user interface that can incur high maintenance costs for your automated suite.

Date and time handling. If the test system can reset the computer’s clocks automatically, then you can automate these tests.

Higher per-test costs and needs for human skills, judgment, and interaction push towards manual testing. A need to repeat tests many times or reduce the cycle time for test execution pushes towards automated testing.

Reasons to Be Careful with Automation

Automated testing is a huge investment, one of the biggest that organizations make in testing. Tool licenses can easily hit six or seven figures. Neophytes can’t use most of these tools—regardless of what any glossy test tool brochure says—so training, consulting, and expert contractors can cost more than the tools themselves. Then there’s maintenance of the test scripts, which generally is more difficult and time consuming than maintaining manual test cases.

The Product Quality Measures

Siva — Tue, 04 Nov 2003 13:51:00 GMT

1. Customer satisfaction index
(Quality ultimately is measured in terms of customer satisfaction.)
Surveyed before product delivery and after product delivery
(and on-going on a periodic basis, using standard questionnaires)
Number of system enhancement requests per year
Number of maintenance fix requests per year
User friendliness: call volume to customer service hotline
User friendliness: training time per new user
Number of product recalls or fix releases (software vendors)
Number of production re-runs (in-house information systems groups)

2. Delivered defect quantities
Normalized per function point (or per LOC)
At product delivery (first 3 months or first year of operation)
Ongoing (per year of operation)
By level of severity
By category or cause, e.g.: requirements defect, design defect, code defect,
documentation/on-line help defect, defect introduced by fixes, etc.

3. Responsiveness (turnaround time) to users
Turnaround time for defect fixes, by level of severity
Time for minor vs. major enhancements; actual vs. planned elapsed time

4. Product volatility
Ratio of maintenance fixes (to repair the system & bring it into
compliance with specifications), vs. enhancement requests
(requests by users to enhance or change functionality)

5. Defect ratios
Defects found after product delivery per function point
Defects found after product delivery per LOC
Pre-delivery defects: annual post-delivery defects
Defects per function point of the system modifications

6. Defect removal efficiency
Number of post-release defects (found by clients in field operation),
categorized by level of severity
Ratio of defects found internally prior to release (via inspections and testing),
as a percentage of all defects
All defects include defects found internally plus externally (by
customers) in the first year after product delivery

7. Complexity of delivered product
McCabe's cyclomatic complexity counts across the system
Halstead’s measure
Card's design complexity measures
Predicted defects and maintenance costs, based on complexity measures

8. Test coverage
Breadth of functional coverage
Percentage of paths, branches or conditions that were actually tested
Percentage by criticality level: perceived level of risk of paths
The ratio of the number of detected faults to the number of predicted faults.

9. Cost of defects
Business losses per defect that occurs during operation
Business interruption costs; costs of work-arounds
Lost sales and lost goodwill
Litigation costs resulting from defects
Annual maintenance cost (per function point)
Annual operating cost (per function point)
Measurable damage to your boss's career

10. Costs of quality activities
Costs of reviews, inspections and preventive measures
Costs of test planning and preparation
Costs of test execution, defect tracking, version and change control
Costs of diagnostics, debugging and fixing
Costs of tools and tool support
Costs of test case library maintenance
Costs of testing & QA education associated with the product
Costs of monitoring and oversight by the QA organization
(if separate from the development and test organizations)

11. Re-work
Re-work effort (hours, as a percentage of the original coding hours)
Re-worked LOC (source lines of code, as a percentage of the total delivered LOC)
Re-worked software components (as a percentage of the total delivered components)

12. Reliability
Availability (percentage of time a system is available, versus the time
the system is needed to be available)
Mean time between failure (MTBF)
Mean time to repair (MTTR)
Reliability ratio (MTBF / MTTR)
Number of product recalls or fix releases
Number of production re-runs as a ratio of production runs

Three Questions About Each Bug You Find

Siva — Tue, 04 Nov 2003 13:54:00 GMT

1. Is this mistake somewhere else also?

2. What next bug is hidden behind this one?

3. What should I do to prevent bugs like this?

Metrics for evaluating application system testing

Siva — Tue, 04 Nov 2003 16:31:00 GMT

Metric = Formula

Test Coverage = Number of units (KLOC/FP) tested / total size of the system
Number of tests per unit size = Number of test cases per KLOC/FP
Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria
Defects per size = Defects detected / system size
Test cost (in %) = Cost of testing / total cost *100
Cost to locate defect = Cost of testing / the number of defects located
Achieving Budget = Actual cost of testing / Budgeted cost of testing
Defects detected in testing = Defects detected in testing / total system defects
Defects detected in production = Defects detected in production/system size
Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100
Effectiveness of testing to business = Loss due to problems / total resources processed by the system.
System complaints = Number of third party complaints / number of transactions processed
Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10
Source Code Analysis = Number of source code statements changed / total number of tests.
Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation
Test Execution Productivity = No of Test cycles executed / Actual Effort for testing

Software QA/Testing - Glossary and Technical FAQs

Siva — Thu, 20 Nov 2003 16:42:00 GMT

http://www.robdavispe.com/free2/

It is a great place for Software QA/Testing FAQ. Great work Rob

Smoke & Sanity Testing

Siva — Thu, 20 Nov 2003 16:53:00 GMT

There are two type of test types, Smoke and Sanity. What they are exactly? Here we go.....

The general definition (related to Hardware) of Smoke Testing is:
Smoke testing is a safe harmless procedure of blowing smoke into parts of the sewer and drain lines to detect sources of unwanted leaks and sources of sewer odors.

It it is related to s/w, the definition is
Smoke testing is non-exhaustive software testing, ascertaining that the most crucial functions of a program work, but not bothering with finer details.

Sanity testing is a cursory testing; it is performed whenever a cursory testing is sufficient to prove the application is functioning according to specifications. This level of testing is a subset of regression testing. It normally includes a set of core tests of basic GUI functionality to demonstrate connectivity to the database, application servers, printers, etc.