More testing help

mattjgilbert — Mon, 15 Dec 2008 17:21:31 GMT

Following on from my post last week with a set of guidelines for testing in the BizTalk world, Michael Stephenson also has a series of posts discussing some of these areas in more detail here. He has also started to look at some of the offerings with BizTalk 2009 which has new support for Unit tests out-the-box.

Shortfalls in testing

mattjgilbert — Wed, 10 Dec 2008 18:04:10 GMT

We’ve been working internally to ensure we have a good story when it comes to testing our Integration solutions. We identified short-comings in our ability to confidently regression test changes and also to capacity plan and predict future growth and resource needs. Too tackle this we have been working on a testing framework for running unit tests, integration tests and load/stress tests using tools like BizUnit and LoadGen. While doing one of our bi-weekly reviews with the development team it struck me that while the foundations to the test framework were in place and were being used, the actual quality of the tests being create and the coverage was really poor. In many cases a simple “pick up the message and does it get there” test was all that existed.

With this in mind I’ve put together a set of guidelines/considerations for our developers for when they are putting tests together. There are some general ones and some BizTalk specific ones and are currently focused more around unit testing and integration testing. We will do another set for performance and stress testing at a later date.

Here’s our first stab at a list of considerations. I’ve not doubt we have missed a lot but we’ll see how it goes.

General Considerations
Separate testing code from the code being tested
Is the unit testing code under source control?
Can all test methods run autonomously so they don’t rely on other test or require that they are run in a particular order?
Is every test able to run over and over again, in any order, and produce the same results?
Make use of both positive and negative testing:
Exercise the code as intended and verify the right result
Deliberately misuse the code and verify correct error handling

Have you included range tests (minimum and maximum values)?
Have you tested cascade effects (other updates and effects on systems other than the direct result/output of the code being tested)?
Have you tested validation rules? (e.g. regex or date validation)
Are the tests self documenting: do they have meaningful names? Do you have a consistent and clear naming convention?
Do test and test fixtures cleanly set-up and take down any test data or configuration they require to run? Do they leave the system in the same state it was prior to tests being executed?
Do you only have single assertions as part of a test?
Are any new test steps you have created generic and reusable in other projects?
Do you need to document custom test steps?
Have you included test data from a range of locales/encoding types (ANSI, EPSIDIC, full unicode) that involved systems may be using?

Specific Considerations
Have you tested every message is delivered correctly?
Have you validated that the delivered message structure is correct?
Have you tested message are delivered to the correct place?
Have you tested each piece of validation logic?
Have you tested valid and invalid inputs and data for validated fields?
Have you tested invalid structures?
Have you tested invalid or missing reference data?
Have you tested message splitting if required?
Have you tested transformations work as expected?
Have you tested orchestrations work as expected?
That they handle any discussions with other systems (e.g. enrichment) consistently?
Compensation and Error handling works as expected?
Transactions work as expected?

Have you tested multiple subscribers receive the message as expected?
Have you written tests for custom components?
Have you tested Pipelines work as expected?

Testing

More testing help

Shortfalls in testing