Alois Kraus

blog

  Home  |   Contact  |   Syndication    |   Login
  114 Posts | 8 Stories | 297 Comments | 162 Trackbacks

News



Article Categories

Archives

Post Categories

Image Galleries

Programming

Patrick Smacchia has shared his thoughts why 100% test coverage is a desirable thing to have. You can test 80% of your code quite easy but the edge cases are the hard ones which take 80% of the time to write good tests for them. Patrick claims that it is worth the effort since in the last 20% you will find most of your bugs. Why is it so hard to get to full coverage? To make the corner cases testable you need some extra ugly constructs in your product (normally via ifdefs where you throw an exception). Many developers do not do it because they feel that this makes their code obfuscated and not maintainable. I am a firm believer in elegant code so lets see how we can keep our code nice and maintainable at the same time.

This made up example is straightforward and it has no way to simulate from outside failures:

                void DoSomeLogic()

        {
            using (Tracer t = new Tracer(myType, "DoSomeLogic"))
            {
                using (var stream = File.Open(tmpFile, FileMode.Open, FileAccess.Read))
                {
                    using (StreamReader reader = new StreamReader(stream))
                    {
                        while (true)
                        {
                            string line = reader.ReadLine();
                            if (line == null)
                                break;
 
                            t.Info(Level.L3, "Got line from file {0}", line);
                        }
                    }
                }
            }
        }

Ok it has some tracing to find out how the code is doing but what if we want to test what does happen if there was an error after the file was opened and when an IOError occurs during reading the second line of the file? If you think I will refactor the code now to make it more testable. Nope not this time. But we need to instrument the code somehow. In a non obvious way we did this already!

Here is the instrumented code:

        void DoSomeLogic()
        {
            using (Tracer t = new Tracer(myType, "DoSomeLogic"))
            {
                using (var stream = File.Open(tmpFile, FileMode.Open, FileAccess.Read))
                {
                    t.Instrument("1");
                    using (StreamReader reader = new StreamReader(stream))
                    {
                        while (true)
                        {
                            string line = reader.ReadLine();
                            if (line == null)
                                break;
 
                            t.Info(Level.L3, "Got line from file {0}", line);
                        }
                    }
                }
            }
        }

Hm the code looks the same except that there is an Instrument trace added. That is strange. Let´s look at the test to simulate an exception after the file was opened:

        static TypeHashes myType = new TypeHashes(typeof(TracerUseCases));
 
        [Test]
        public void Inject_Fault_After_File_Open()
        {
            DoSomeLogic();
            TracerConfig.Reset("null");
            Tracer.TraceEvent += (severity, typemethod, time, message) =>
                {
                    if (severity == Tracer.MsgType.Instrument)
                        throw new IOException("Hi this is an injected fault");
                };
 
            Assert.Throws<IOException>(() => DoSomeLogic());
        }

And here is the test to force an exception during reading the second line of a file:

        [Test]
        public void Inject_Fault_During_Stream_Read()
        {
            TracerConfig.Reset("null");
 
            int lineCount = 0;
            Tracer.TraceEvent += (severity, typemethod, time, message) =>
            {
                if (severity == Tracer.MsgType.Information && message.Contains("Got line from"))
                {
                    lineCount++;
                    if (lineCount == 2)
                        throw new IOException("Exception during read of second line");
                }
            };
 
            Assert.Throws<IOException>(() => DoSomeLogic());
        }
 

The trick is simple but powerful. We can delegate the aspect fault injection to the already existing tracing calls which we do intercept in our tests and throw an exception when needed. The best thing about this is that you need no more changes to your product code to inject faults at critical points in your application. You only need to add some trace statements at strategic points which is a good idea since you want to know what is going on in these cases anyway. You can download the ApiChange Tool Api here. To try it out you can simply reference the ApiChange.Api.dll and copy some of the above code into your solution and you have full tracing enabled. To configure tracing you can look at some documentation here.

When something unexpected happens you can simply change the Tracing configuration in your unit test from null to console to see the tracing output in your test runner.

***** UnitTests.Infrastructure.Diagnostics.TracerUseCases.Inject_Fault_During_Stream_Read
23:34:11.712  05876/04208 <Instrument > UnitTests.Infrastructure.Diagnostics.TracerUseCases.DoSomeLogic 1
23:34:11.713  05876/04208 <Information> UnitTests.Infrastructure.Diagnostics.TracerUseCases.DoSomeLogic Got line from file Line 1
23:34:11.714  05876/04208 <Exception  > UnitTests.Infrastructure.Diagnostics.TracerUseCases.DoSomeLogic Exception thrown: System.IO.IOException: Exception during read of second line
   at UnitTests.Infrastructure.Diagnostics.TracerUseCases.<>c__DisplayClass5.<Inject_Fault_During_Stream_Read>b__3(MsgType severity, String typemethod, DateTime time, String message)    at ApiChange.Infrastructure.Tracer.TraceMsg(String msgTypeString, String typeMethodName, DateTime time, String fmt, Object[] args)
   at ApiChange.Infrastructure.Tracer.Info(Level level, String fmt, Object[] args) 
   at UnitTests.Infrastructure.Diagnostics.TracerUseCases.DoSomeLogic()
   at UnitTests.Infrastructure.Diagnostics.TracerUseCases.<Inject_Fault_During_Stream_Read>b__4()
   at NUnit.Framework.Assert.Throws(IResolveConstraint expression, TestDelegate code, String message, Object[] args)

23:34:11.724  05876/04208 <         }}< UnitTests.Infrastructure.Diagnostics.TracerUseCases.DoSomeLogic Duration 13ms

If you need to add traces for the sole purpose to inject faults into your product code you can use the Tracer.Instrument overloads which are conditionally compiled into your code when you add INSTRUMENT to your conditional compilation constants to your project. This way you get the best of both worlds. No code in your release build but full instrumentation where necessary.

Have fun getting your code to 100% coverage.

posted on Monday, July 19, 2010 9:42 AM

Feedback

# re: Fault Injection Done Right 8/8/2010 7:03 AM John f
Is there supposed to be a call to DoSomeLogic in the first line of the first Inject_Fault...()?
I can't see why it's there, and it's not in the 2nd example.

# re: Fault Injection Done Right 8/22/2010 9:41 AM Alois Kraus
Hi John,

this can be called bad test coding style. The first test checks if the method DoSomeLogic does not throw any exception at all if tracing is disabled.
Then the second test skips this "test" and causes an exception in DoSomeLogic while reading from the stream.
I should have refactored it out into a separate test to make it more clear but at the other hand it saves some typing ...

Good catch.

Yours,
Alois Kraus


Post A Comment
Title:
Name:
Email:
Comment:
Verification: