Geeks With Blogs
Maryanne's Blog Another Geek with A Blog

WARNING--THIS IS A VERY--VERY--LONG POST

To respect a friend's blog space, I've decided to respond to a commenter on Theo Moore's blog  regarding a review he posted about TestComplete.

The commenter posted the following comment to me:


Dear MES,

Thanks for answering my query on be half of Theo. I would still love to hear from her (I think theo is female not a male).

>> there are things that you can do with QTP that qualify as "testing."

Can you name them? I am especially interested to know what is testing according to you. QTP is a tool, a software that can help a human tester in executing some tests and do some comparision. Testing in my opinion is a deep human intellectual investigative activity that happens to cognitive engagement of human with the environment of software. It is way beyond what a tool can do. It is possible that testing according to you may be different from my definition. But, equating automation to testing is something that is difficult for me accept with some probing around.

>>> It is capablye of so much more than simple record and playback

Yes it is ... I did not comment on record/playback aspect. QTP can do lots of things. None --- none of them can qualify to be testing in totality. QTP helps a human tester in testing. I agree it does more than simple R/P

>>> I've used it to verify applications are storing data to a database correctly using ADO DB connection objects. I've used the DOM model of a webpage to dynamically determine objects on the page (with a little help from my friends of course) so that the tests were not relying on the Object Repository.

You are right. QTP can do all these. But none of these can be called "testing" on their own. They are tasks or constituents of testing. If you call these as testing - I can only say that your definition of testing is very narrow and is nothing more than comparing expected result to actual result.

>>> So, frankly--castigating Theo on his choice of language in an industry with no standards regarding what is a Test Case versus Test Script versus Test Scenario only shows your ignorance

I may be ignorant -- but I am open to learn from you and improve my understanding of testing. I feel that what theo is calling as testing is actually is automation. I am sure she knows the difference.

By saying there is no standard in terminologies, what stops me from calling automation as fish and testing as elephant. Blogs and public communications are open for criticism and scrutiny. So, I posted my view points - let Theo clarify what is the meaning of testing according to her.

>>> is in no way suggestive of Theo's lack of experience in this arena. Which btw he has considerable knowledge.

My intention was never to question and challenge Theo's experience. As a tester I challange anything that is testable.

Even my this comment is open for challenge

Shrini Kulkarni 
First--I feel M. Kulkarni should understand a few things about my Quality Control/Quality Assurance (QA/QC) background before I respond to his questions directly.
  1. I have been a QA/QC professional since 1998. 
  2. I am a Certified Software Testing Engineer (CSTE awarded in 2005)
  3. In my career I have been an accomplished manual tester.
  4. Currently I consider myself to be an accomplished software automation engineer specializing in Keyword driven automation framework development using HP QuickTest Pro as my toolset with homegrown extensions via vbScript.
  5. And since he seems to think it matters--I am female.

For the purposes of this discussion I am going to assume that M. Kularni is referring to Software Testing.

Now to respond to M. Kulkarni's questions:

>>I am especially interested to know what is testing according to you.

Software testing is a deep and complex subject. 

There are many activities that qualify as testing.  In the strictest terms, testing is a Quality Control mechanism to ensure software defect detection and correction.  Now, does that mean that all "tests" are executed against software.  No--it does not. Any time a Quality Control or Quality Assurance professional performs a document inspection on a business requirements document, software specification or technical design, they are executing "tests" against those documents.  They are verifying that the document is complete, that the requirements listed within are clear, consise, measurable, testable, unambigous and without room for interpretation.  This is something that an automated testing tool could never do.  Well, I say never, but until we create artificial intelligence, it may as well be never.  These are the types of activities that humans do best.

That doesn't mean that Automated testing doesn't have a roll to play in a software testing organization.  The best use of automated testing is to do those tasks that are frankly monotonous.  Those tasks that when performed by people are likely to produce user error.  For example, I've been spending the last two weeks at my job automating a series of test cases that a manual tester has to execute every week.  These tests are rudimentary, very simple and frankly boring.  They are the kinds of tests that in my 10 years of experience have been "passed" by manual testers without them actually doing the work because they've ran them week after week after week with no failures.

This is the strength of an automated test suite.  It does the work that people don't want to do.  In a parallel sense, Babbage invented the calculator because people didn't want to add things anymore.  Its error prone and frankly machines do those kinds of things better.

 >>Testing in my opinion is a deep human intellectual investigative activity that happens to cognitive engagement of human >>with the environment of software.

That is an interesting point of view.  I don't know that we differ on our definition of testing.  My educational background is in the hard sciences, Physics in particular.  And my approach to software testing is unique in that when performing manual testing activities, be it test script execution or exploratory testing, I take a very scientific methodical approach to testing.  The principal difference between software testing and scientific method experimentation is that in Scientific Method, if your experimental results don't match your theory you change your theory.  In software, you change your experiment.  What do I mean by this:  Well in software the "theory" is your requirements document.  And the experiment is what the software does when you try to apply a set of steps to recreate the "theory" as described in the requirements document.  If the software doesn't recreate the theory, you change the software to make it match the theory.

So what does that mean in regards to automated testing.  Well...first and foremost, automated testing cannot be done on "new" code.  A manual tester must ensure that the code is working correctly and that the code is stable before you consider applying an automated solution.  Meaning..."Regression Testing" should be the primary domain of automated test execution.  The exception to this rule would be using an automated solution to create data for testing a new function.  An example would be generating rows of data in a database to verify an ETL process works correctly.  So,  using an automated script, a tester puts 100 rows into a transactional system, then requests the DBAs run the ETL and see how many rows end up in the data warehouse.  It would be extrememly time consuming for the tester to create this data themselves.  So, using an automated process allows the tester to exercise the code that changed..without spending manhours putting the seed data in the system first.

So I think this gets the ball rolling regarding how I feel Automated Testing plays a roll in a QC/QA organization.  Is it a panacea?  No... Is it capable of everything?  No

However, does it have its place within a testing approach?  Absolutely.

Thanks for reading this exhaustive post..I'm sure there's more to come. 

Posted on Thursday, November 20, 2008 8:52 PM General QA/QC Topics | Back to top


Comments on this post: Can Automated Testing be Called "Testing"?

# re: Can Automated Testing be Called "Testing"?
Requesting Gravatar...
I also wanted to respond to what I read which was posted by Mr. Kulkarni.

It is apparent that Mr. Kulkarni's exposure to automated testing has been very limited. The points being made really sound like reactions to a very high-level demo performed by a vendor, as opposed to in-depth analysis of what some of these tools actually are capable of.

There are two things going on here:

1) Automation (this is what Mr. Kulkarni is speaking of).

2) Testing (this is where you actually MEASURE something in the application).

I've seen several shops that really just do automation. This is the "entry" layer for most tools. Simply put, you have to get from point A to point B before you can validate, or "test", anything. Too many times inexperienced automated testers stop here.

Does automation alone have benefits? Sure. Such as in situations where you are doing data setup, or simply validating navigation is working properly.

Mature automated testers also realize that automation is really only the first step. Once you have automation in place, you need to actually automate the "testing" that is supposed to take place.

This comes in the form of "checkpoints", as well as custom validation points the tester codes. For example: Colors of text, button state, default radio/checkbox selections, style code, image existence or placement, hidden objects, variable validation, text values, querystring parsing, database comparisons, page load time, source code analysis, etc. Essentially you can check just about any property on any "object" on the page. Just as a live person would do.

The most difficult part of this process is to get your manual testers to actually think about what it is they do when they test, and document it. What are they looking for to consider a test a "pass"? These are the things your automated script must also validate.

Have something your automated script can't validate for some reason? Simple, perform a screen capture and have your manual testers review the automated results after the run.
Left by Norm Wall on Apr 28, 2009 6:24 PM

# re: Can Automated Testing be Called "Testing"?
Requesting Gravatar...
I dont understand what Shrini is trying to prove here. QTP of course is a testing tool, it is not just a recording or lpay back tool. You feel it because you are not able to synchronise automation with manual thoughts. I dont think Theo is worng in anything she has told.
Left by Sandeep on Jul 06, 2009 5:44 AM

# re: Can Automated Testing be Called "Testing"?
Requesting Gravatar...
Shrini Kulkarni. Grow up.

The blog posted by Theo Moore answers one of critical questions that many today want to know. I think you just want to prove how good and great you are with you theory. I relate you with 3 idiots movie character "Chatur"

Grow up and learn to appreciate. You may be welcomed to discuss but not to do an offtopic debate using your jargons.
Left by MKM on Jul 23, 2010 4:11 PM

# re: Can Automated Testing be Called "Testing"?
Requesting Gravatar...
I very much appreciate all the information you and Theo provided about QTP vs TC.
Left by MG on Dec 16, 2010 3:45 PM

Your comment:
 (will show your gravatar)


Copyright © Maryanne Sweat | Powered by: GeeksWithBlogs.net