April 2004 Entries
Many people and organizations are confused about the difference between quality assurance (QA), quality control (QC), and testing. They are closely related, but they are different concepts.
But all these three are useful to manage risks of developing and managing software.
- Quality Assurance: A set of activities designed to ensure that the development and/or maintenance process is adequate to ensure a system will meet its objectives.
- Quality Control: A set of activities designed to evaluate a developed work product.
- Testing: The process of executing a system with the intent of finding defects. (Note that the "process of executing a system" includes test planning prior to the execution of the test cases.)
QA activities ensure that the process is defined and appropriate. Methodology and standards development are examples of QA activities. A QA review would focus on the process elements of a project - e.g., are requirements being defined at the proper level of detail.
QC activities focus on finding defects in specific deliverables - e.g., are the defined requirements the right requirements
Testing is one example of a QC activity, but there are others such as inspections
The difference is that QA is process oriented and QC is product oriented.
Testing therefore is product oriented and thus is in the QC domain. Testing for quality isn't assuring quality, it's controlling it.
Quality Assurance makes sure you are doing the right things, the right way.
Quality Control makes sure the results of what you've done are what you expected.
Source:
http://c2.com/cgi/wiki?QualityAssuranceIsNotQualityControl
http://www.mosaicinc.com/mosaicinc/rmThisMonth.asp
Here is the greate post on Test Documentation by James Bach.
Source: James Bach's Blog: Software Testing and Quality Archives
A lot of people I teach seem to be under pressure to create more documents to describe their test process. But documenting is not testing. It is one of the chief distractions to testing.
"James Bach hates documentation!", some people will say. No, I don't. I hate waste. I hate boring clerical work that unnecessarily interrupts productive and interesting intellectual work. You should hate it too.
I'm a writer for cryin' out loud. I like documentation-- when it is the solution to a problem that matters, and not merely a management obsession. But if you're trying to move to concise test documents (outlines, matrices, one-page reference sheets, and other minimalist formats) you may need help persuading management and co-workers.
Here are some ideas:
- Show management how much less testing we are able to do because we are spending so much time with documents.
- Show management how certain kinds of testing isn't done at all just because it is hard to document (exploratory testing and complex scenario tests often fall in this category). This is perhaps the most chilling effect of over-documentation, especially in the realm of medical devices. I keep seeing medical device test doc that is simplistic, in all its obesity, to the point of near worthlessness.
- Examine closely what testers are doing and show that they aren't even following the documentation (often they aren't, in my experience as a consultant who audits test processes).
- Demonstrate the power of exploratory testing (a less heavily documented approach). One day of ET is often sufficient to find what would take a week to find when following detailed documented test procedures.
- Demonstrate the value of concise test documentation (matrices, outlines).
- Consider documenting at the level of test activities rather than test cases.
- Consider automatic documentation (via log files produced by the product under test or via an external logging tool such as Spector).
- Ask the question: what exactly are we getting from our documentation? Don't accept any theoretical answers. For example, one typical answer is that documentation protects the company from the ill effects of staff turnover. But does it? Probably not, in my experience. That's a theory based on ignorance about how people learn. In real life, new testers learn what they need to know by playing with the product itself, and talking to the people around them. In my experience, testers come up to speed in a few days at most. And in my experience, test documentation is often of such poor quality that it's better ignored than followed. You have to go by your own experience, of course. I'm just suggesting that you ask the questions and make your own observations.
Heavy documentation is often a consequence of managers and testers who just aren't thinking through the reasons why they do things. They hear that documentation is good, but they don't stop to consider the cost of documentation, or watch how documentation is actually used (or more often, ignored).
Myth #1
Myth:
- OO testing is unnecessary. OO promotes incremental development and reuse, so we have a more effective way to develop trustworthy classes.
Reality:
- Human error is as likely as ever. We have to check class functionality and interactions between classes
Myth #2
Myth:
- Testing is a structured waterfall idea and isn’t consistent with incremental OO development
Reality:
- Tests can be designed and exercised at many points in the process
- Paradigm of “design a little, code a little” becomes “design a little, code a little, test a little”
- Extreme Programming (XP) promotes early testing
Myth #3
Myth:
Reality:
- Pay me now or pay me MUCH more later
- Failures in operational systems can cause severe secondary problems
- Proper testing is very cheap by comparison, even when done manually
- Efficient testing of large or complex systems needs some automated support
Myth #4
Myth:
- OO testing is the same as conventional software
Reality:
- Polymorphism, inheritance, encapsulation present opportunities for error
Myth #5
Myth:
- Conventional testing is useless for objects
Reality:
- There is a large body of knowledge about testing
- Basic testing techniques continue to apply with necessary changes
- If scope of testing is small, ie classes and collaborations
- Testing techniques are very OO specific
- As scope of testing gets larger
- Testing techniques become more traditional
Myth #6
Myth:
- Inheritance means never having to say you are sorry
- Specializing from a trusted superclass means the inherited part of subclasses will also be correct
- We don’t need to retest inherited features
Reality:
- Subclasses create new ways to misuse inherited features
- Different test cases are needed for each subclass
Myth #7
Myth:
- Reuse means never having to say you are sorry
- Reusing trusted class means behavior of server object is trustworthy and doesn’t need to be tested
Reality:
- Nothing prevents a new client class from using the server object incorrectly
- All client class use of a server needs to be exercised
Myth #8
Myth:
- Black box testing is sufficient
- Designing test cases using class interface and specification assures the class is fully exercised
- White box testing violates encapsulation
Reality:
- Studies show black-box test suites typically only exercise 1/3 to 1/2 of the statements (let alone paths or states)
- Typically miss abnormal paths, exception and error handling
If 'B' represents what you need in order to perform satisfactory test automation for your own particular context and 'A' is the set of tools that are available to you, then "Homebrew Test Automation" is what you may need to create on your own in order to bridge the gap between 'A' and 'B'.
"Homebrew Test Automation" may include adding extensions to or modifying existing tools. It may also include writing new tools and/or integrating exiting tools in novel ways.
Whether the tools are commercial or free should be irrelevant. What is relevant is how easily the tools lend themselves to the above activities, either through providing source code, a well thought out extensibility interface or some combination, and whether you can ultimately demonstrate a positive return on investment in time and materials.
--written by Mike Silverstein in Stickyminds Roundtables