I was poking through Jason Haley's blog today when I came across one of his interesting links (which are usually pretty interesting btw). It pointed to a post by Phillip Haack titled A Testing Mail Server for Unit Testing Email Functionality. I generally enjoy reading about other’s unit testing experiences as I often gain quite a bit of perspective (and get to see a lot of problems I may not otherwise get to see).
Basically what he has done is taken an open source SMTP server and used it for in order to allow his unit tests to check to see if an email that was sent through the .NET email libraries actually arrived at its sender. Let me say that he has come up with a very inventive way of automating this integration test. I personally think that his code is extremely useful for automated integration testing. I have to admit, I have done automated integration tests for emailing and I used a far less elegant solution.
As I was reading through the post I came across the comment.
As for the semantic arguments around whether this really constitutes an Integration Test as opposed to a Unit Test, please don’t bore me with your hang-ups. Either way, it deserves a test and what better way to test it than using something like MbUnit or NUnit.
Well, let me start by saying it is an integration test at best (I might even classify this as a user acceptance type test) and has no place being a unit test.
He did however hit the appropriate unit test on the head earlier in his post when he suggested an EmailProvider (I say EmailService but that truly is semantics) which could then be mocked for testing purposes. I would personally either create my own abstraction of an “Email” class or use the one from System.Net.Mail as a parameter instead of passing parameters but the reasons for this will have to wait for another post.
That said let’s look at the test case he created to test that email actually got delivered.
DotNetOpenMailProvider provider = new DotNetOpenMailProvider();
NameValueCollection configValue = new NameValueCollection();
configValue["smtpServer"] = "127.0.0.1";
configValue["port"] = "8081";
TestSmtpServer receivingServer = new TestSmtpServer();
"Subject to nothing",
"Mr. Watson. Come here. I need you.");
// So Did It Work?
ReceivedEmailMessage received = receivingServer.Inbox;
I can identify a few places that could cause this test to fail that would not cause our mock to fail otherwise.
1) We did not properly setup our configValue["smtpServer"] configuration
2) We did not properly setup our configValue["port"] configuration
3) Someone is already listening on our configured port
4) We did not properly start our testing server or it is failing in some way (i.e. configuration)
5) We have a firewall that prevents the communication from working between the client and our email server
6) There is no accessible network between the client and the server
7) The Ethernet elves ate the address that the mail was sent to
8) Numerous other environmental factors
Are you noticing a pattern in these items? We could have configuration failures, failures in our test code, or environmental failures; not a single case is actually testing our code beyond what a mock would do. When our test fails it only tells us that the environment that the test was run in was not appropriate for the test code and as such the test failed. I thought that unit tests were supposed to test your code?
The test results are not dependent upon the code in question; they are dependent upon the testing environment.
If we were to try to come up with an analogy for the type of unit test this was we could say it is like testing your car by turning it on. When the car turns on we can have a grand celebration that our car did in fact turn on, but what if the car does not turn on. All we have gained from this test is that the car did not turn on; we have no insight as to why the car did not turn on.
Any time we have a unit test which gives us no insight as to why a failure is occurring we need to re-think the value of the unit test.
What if the test succeeds, does that tell us our code works? No it tells us that the email client we used was compatible with our test email server and that our testing environment was setup appropriately to run the test.
Unit tests are overhead for a system. They are no different than documentation; they need to pull their weight in order to stay around. Let me go back to the quote statement saying that this does not actually deserve a test. This particular test carries with it a lot of baggage,
1) You have to have the email server library included/maintained with your project
2) You have to have a system ready to run the email server (i.e. nothing running on the port it uses)
3) you have to have an environment where you won’t have firewalls
4) You have to have TCP running (I know this is pretty common but it is still a prerequisite)
Given all of this baggage and the fact that it doesn't really test anything.. I say that this test does not deserve to be around.
What is really scaring me is how this test came into being a unit test using TDD. TDD would have stopped this test ever appearing. Where’s the red bar? The only way to make the test fail where a mock would not providing you have a working method of sending an email (which was said in that they are using System.Web.Mail) is to change the test or the environment.