Dylan Smith

My team is going to start using the Manual Testing functionality available in VS 2010 for one of our larger projects.  We started today to migrate some of our manual test scripts over to Test Cases/Test Plans in Test and Lab Manager.  We ran into a problem immediately that almost prevented us entirely from continuing to use the product.

If you have a Test Case with a lot of Test Steps the scrolling in the Test Case editor in Test and Lab Manager is broken.  When you get enough Test Steps so that you have to scroll vertically to see them all you’re in trouble.  The problem is that the scroll bar keeps resetting itself to the top every time you click in the Test Steps grid.  So for example if you scroll down to see one of the Test Steps further down the list and click it to edit it the scroll bar brings you right back to the top and you can’t see what you’re doing.  The same happens when you try to add new Test Steps to the end of the list, you can still add items but you have to type “blind” since the scroll bar brings you back to viewing the top of your list.

Luckily a quick phone call to Aaron Kowall provided me with a workaround.  Every Test Case is stored as a Work Item, so you can use the Work Item editor in Visual Studio to edit your Test Cases.  The scrolling issue is only present in Test and Lab Manager and not Visual Studio as far as I can tell.  Since I figured this out I’ve been creating and managing my Test Cases in T&L Mgr still, but for any Test Case with lots of Test Steps I jump over to VS to finish inputting the Test Steps or do any editing.

I was trying to create a Sysprep’d VPC image containing VSTS 2010 Beta 2 + TFS 2010 Beta 2 to distribute to the rest of my team to try out some of the new features.  Unfortunately, distributing a Sysprep’d image means that everybody will input a new computer name when they boot it up the first time.  TFS doesn’t like it when you go and change the computer name on it.  I found an MSDN article talking about how to move a TFS server from one domain to another that probably contains the proper steps to fix up TFS once you’ve set the computer name.  But that process looks fairly complex, and the whole point of making these VPC images was to make things simple for my team members who wanted to try out Beta 2.  I’m thinking that creating a VPC without TFS on there and having them install it themselves will be easier than having them try and fix a broken install of TFS due to a changing computer name. Looks like I’m going to be spending the evening creating a VPC image….again.

I just went through the process of creating a VHD for use by myself and some other members on my team.  I had a pre-built base image with Windows Server 2008 and SQL Server 2008 Enterprise that I used to begin the process (Note: I was hoping to use Windows Server 2008 R2 but it appears it only comes in 64-bit and VirtualPC doesn’t support 64-bit).

From there I installed Visual Studio 2010 Ultimate Beta 2 and Team Foundation Server 2010 Beta 2.  Everything went extremely smoothly except for one small gotcha right at the end.  After installing TFS I was going through the configuration wizard and trying to configure Team Build.  The first time I ran through the config wizard it gave me an error on the last step which was something like initializing the team build service.  The error message said something about not finding the user account, which was odd since I just used the local Administrator account and made sure to click the “Test” button in the config wizard to check that the username/password I entered was valid.  The exact error message was:

TF255070: Cannot register Team Foundation Build Service Host: User Account localhost\Administrator was not found

After a bit of searching and playing around I figured out how to fix the problem.  When running through the TFSBuild configuration wizard one of the steps is pointing it to the Team Collection to use.  When you do this and you browse for your TFS instance, and since this is probably the first time doing this you’ll have to go into the TFS Servers dialog and add in the instance, make sure that you use the actual machine name and not localhost or an ip address.  My mistake was I just entered localhost.  When I ran through the wizard a 2nd time (actually it was more like the 8th time before I figured this out) and used the actual machine name everything else worked great.

Next up I have to sysprep this and distribute it to my team.  I’m hoping that the Sysprep and distribution of the VM goes smoothly without causing any TFS issues.

So the other day at work we were doing an architecture/design review on a project to share knowledge gained with other project teams.  After making the code available one of the other architects came to me with a specific piece of the code in hand asking us to justify why it seemed so complex (in this case the comment was mainly because the code used lambda's generously and he had limited exposure to them previously).  Here is the code in question:

Original Approach

public void Save(IJob job)
{
    this.AutoCommit(s => s.SaveOrUpdate(job));
}

public static void AutoCommit(this IDisplacementStorage storage, Action<IStorageSession> action)
{
    storage.AutoSession(s =>
                            {
                                using (var tx = s.BeginTransaction())
                                {
                                    try
                                    {
                                        action(s);

                                        tx.Commit();
                                    }
                                    catch
                                    {
                                        tx.Rollback();

                                        throw;
                                    }
                                }
                            });
}

public static void AutoSession(this IDisplacementStorage storage, Action<IStorageSession> action)
{
    using (var session = storage.OpenSession())
    {
        action(session);
    }
}

I wanted to compare this implementation to some more simplistic implementations.

Option 1

public void Save(IJob job)
{
    using (var session = this.OpenSession())
    {
        using (var tx = session.BeginTransaction())
        {

            try
            {
                session.SaveOrUpdate(job);
                tx.Commit();
            }
            catch
            {
                tx.Rollback();
                throw;
            }
        }
    }
}

Aside from clearly breaking the Single Responsibility Principle the Option 1 approach is less code (8 lines in one method vs 11 lines in 3 methods) and also easier to understand/read.  However the SRP problems become more evident once you start to flesh out the storage layer and need to implement more Save methods.  Lets say we needed a Save(IZone), Save(IRoom), Save(IDiffuser), etc.  We could just copy/paste the code and change the argument type, but that would quickly result in much more code.  The Original Approach only requires 1 extra method containing one line of code for each additional Save method.  The Option 1 approach would require an extra method containing 8 lines of code for each additional save method.  It also results in lots of code duplication between Save method, so if you wanted to change the way you create sessions/transactions, or add logging, or change the error handling, or anything like that you have to make changes in many places as opposed to just one in the Original Approach.  An obvious answer to some of these concerns is to refactor out the common code.  See Option 2 below.

Option 2

public void Save(IJob job)
{
    SaveOrUpdate(job);
}

public void Save(IZone zone)
{
    SaveOrUpdate(zone);
}

public void SaveOrUpdate(object target)
{
    using (var session = this.OpenSession())
    {
        using (var tx = session.BeginTransaction())
        {

            try
            {
                session.SaveOrUpdate(target);
                tx.Commit();
            }
            catch
            {
                tx.Rollback();
                throw;
            }
        }
    }
}

The SRP violations are getting better now, the Save method clearly only has one responsibility now, although the SaveOrUpdate is still taking on multiple responsibilities (creating/disposing session, creating/committing/rolling-back transaction, saving the object).  The Option 2 approach now has 9 lines of code spread across 2 methods – still better than the Original Approach in this respect.  Adding additional Save methods now simply requires adding a new method with a single line of code, which is the same as in the Original Approach.  However, what about deletes?  To handle that we could create a similar method as the SaveOrUpdate but to handle deletes; however, that would be another 8 lines of code and an additional method for deletes.  What about more complex scenarios, for example lets say it was a financial application and you had a TransferFunds method that needed up to update 2 account balances within a transaction.  In that case or other cases that are more than just a simple save/delete on a single object we’ll need to essentially copy/paste the code to create/dispose the session + transaction each time along with the appropriate try/catch block.  A nicer way to handle arbitrary scenarios like this is instead of hardcoding the action to be taken we allow the caller to pass in an Action that contains the scenario-specific data-access code.  This might look something like Option 3 below.

Option 3

public void Save(IJob job)
{
    ExecuteInTransaction(s => s.SaveOrUpdate(job));
}

public void Delete(IJob job)
{
    ExecuteInTransaction(s => s.Delete(job));
}

public void TransferFunds(IAccount fromAccount, IAccount toAccount)
{
    ExecuteInTransaction(s => 
                              {
                                  s.SaveOrUpdate(fromAccount); 
                                  s.SaveOrUpdate(toAccount);
                              });
}

public void ExecuteInTransaction(Action<IStorageSession> action)
{
    using (var session = this.OpenSession())
    {
        using (var tx = session.BeginTransaction())
        {

            try
            {
                action(session);
                tx.Commit();
            }
            catch
            {
                tx.Rollback();
                throw;
            }
        }
    }
}

The lines of code required for just the save is the same as in Option 2 (9 lines of code spread across 2 methods) but now we have much more flexibility to do other data-access scenarios such as delete or more complex use cases with minimal code required.  The ExecuteInTransaction class arguably still violates the SRP because it is responsible for handling the session creation/disposition as well as handling the transaction.  We could break this out to satisfy SRP; but a more reasonable justification would be that you might want to be able to execute data-access code outside of a transaction, or you might want to directly manage the transaction object rather than the simple transaction handling baked into the ExecuteInTransaction method.  We could do this by refactoring out the session handling code into it’s own method.  We can follow the same pattern we did with using the Action in the ExecuteInTransaction method to allow the caller to pass in arbitrary actions to be executed in the session.

Option 4

public void Save(IJob job)
{
    ExecuteInTransaction(s => s.SaveOrUpdate(job));
}

public void Delete(IJob job)
{
    ExecuteInTransaction(s => s.Delete(job));
}

public void TransferFunds(IAccount fromAccount, IAccount toAccount)
{
    ExecuteInTransaction(s => 
                              {
                                  s.SaveOrUpdate(fromAccount); 
                                  s.SaveOrUpdate(toAccount);
                              });
}

public void ExecuteInTransaction(Action<IStorageSession> action)
{
    ExecuteInSession(s =>
                        {
                            using (var tx = s.BeginTransaction())
                            {
                                try
                                {
                                    action(s);
                                    tx.Commit();
                                }
                                catch
                                {
                                    tx.Rollback();
                                    throw;
                                }
                            }
                        });
}

public void ExecuteInSession(Action<IStorageSession> action)
{
    using (var session = this.OpenSession())
    {
        action(session);
    }
}

If we rename ExecuteInTransaction -> AutoCommit and rename ExecuteInSession -> AutoSession we now have the exact code we started off with in the Original Approach (except that the Original Approach implemented them as extension methods whereas Option 4 is written as instance methods directly on IDisplacementStorage).

So as I mentioned in my last post I have this large-ish .Net application which has a pretty low quality level.  The first task was to convince the powers that be that there is sufficient benefit to be had to justify the cost of dedicating 1-2 developers to focusing on quality improvements.  Now we need a plan for how to approach making quality improvements.

Ultimately I think we are going to need to do some major work refactoring large parts of the application (hopefully we can break these down to perform them incrementally), but before we even consider that there are some other things that we can and should do first.  For starters we currently have no automated tests in place.  We have some manual test scripts that we perform on a routine basis to check for regressions, but even those are fairly limited in how much of the functionality they actually exercise.  So an obvious place to start it to work on putting a suite of automated tests in place with the goal of enabling us to refactor with some level of confidence that we didn’t break the rest of the application.

Writing the automated tests may prove a little tricky since this application wasn’t written with testability in mind.  The “seams” that Michael Feathers talks about in his book - areas where you can decouple pieces of behavior to test them in isolation - are simply not available in the current code-base to allow tests to be written in the way that they would be if this application was written with testability in mind (using TDD for example).  So to begin my plan is to write tests from a high-level by attempting to use a UI automation framework such as NUnitForms.  This should allow us to introduce some very broad tests that exercise the entire application stack to begin creating that safety net so we know when a code change affects the application behavior.  We plan on using code coverage metrics to track our progress in this activity.  I’m going to set some rather arbitrary goal to begin with – say 40% coverage – that we’ll attempt to achieve across all the assemblies.  Once we reach that we’ll start to identify some good candidates for refactoring, and focus any new tests on those specific areas of the code with a higher goal for code coverage – maybe 60% for the specific assembly we’re targeting.  At this point we can start performing some of the refactoring work using the tests as our safety net to give us some level of confidence that we aren’t destabilizing the codebase with the changes we make.

** I’m aware that code coverage stats alone don’t represent whether you have a useful test suite or not, but they do give a convenient quantitative measure you can use to track progress.

In parallel to the exercise of building up a test suite I’m planning on making use of some static analysis tools to help improve quality at a low-level.  The first thing we’re going to do is go through and fix up all the compiler warnings we currently have.  Once we get that cleaned up we’ll update the build to fail on warnings to prevent any from getting re-introduced.  Next on the list is starting to apply the Visual Studio Code Analysis rules (aka FxCop).  The plan is to apply one rule at a time, go through the entire code-base and fix up all violations, then update the build to apply that build and fail on violations.  Doing this one rule at a time is a must since there are literally tens of thousands of violations in the code-base, going one rule at a time at least allows us to do it somewhat incrementally.  Once the FxCop rules are all in place next up is starting to apply StyleCop rules in the same manner to introduce some style consistency across the code-base.  After StyleCop we might move on to using NDepend and using the CQL (Code Query Language) built-in rules as well as defining some of our own to help encourage proper use of the various components/layers within our application.

Check back in a while and I’ll update you guys on how these activities went, whether we realized any significant value, and any lessons learned that may help other people out in a situation.

Also, if anybody has any experience using UI Automation frameworks for writing tests at the UI level I’d be interested to know what framework you used, what you thought of it, and if you were using Continuous Integration any issues you ran into and you worked around them (for example, I’ve heard that doing UI Automation testing from a build server can cause all kinds of issues if there are any modal dialogs going on due to the fact that most build servers run as a windows service).

So I started my new job at Anvil Digital a few weeks ago.  One of the first things I’ve been doing is just taking a look at the current projects and processes and finding some opportunities and/or weaknesses where improvements can be made.  One of the projects under-way is the development/maintenance of a large-ish .Net application.  When I took a look under the covers at the design/architecture I noticed there is clearly a lot of room for improvement.  In this specific instance, what has happened is the application has grown from several smaller applications that along the way have been merged together and grown drastically beyond what the original developers expected.  Compounding the problem is that there has been no real strategy in place for the application design or architecture, so each of the 8-10 developers working on the application has gone off and done their own thing when it comes to design.  Some of the approaches taken are reasonably well implemented, some not so much; but the key point is that there is little to no consistency between the various areas in the code-base.

I’ll go into a bit more detail in what some of the quality problems as I see them are in a minute.  But what I’m really interested in discussing in this post is what is the cost of having poor internal quality?  How do you justify the cost of taking time to improve quality through refactoring or other methods?  I have some thoughts on this that I’ll share, but first let me give some examples of the quality weaknesses as I see them for this specific application:

• The application doesn’t take advantage of OO really at all.  The key concepts in the application do not have corresponding classes to represent them.  There is very little attention paid to having cohesive classes that encapsulate specific behavior and data.  The abstractions/concepts are leaking all over the place.
• The various components that make up the application are not layered very well.  For example there are many cases of circular dependencies that make compiling the application very tricky at times.
• There are many cases of “god classes”.  I’ve seen single classes that have over 12,000 Lines Of Code.  Definitely not following the Single Responsibility Principle
• Another symptom of the poor quality is the fact that building the application throws 500+ compiler warnings, and I think some of those are just saying that the max warnings threshold has been reached, so there are probably many more not being reported.

So what is the cost of having poor internal quality like this?  I know there’s been some discussion recently in the blogosphere by Joe Rainsberger on the Speed / Quality Barrier, Ron Jeffries on Quality-Speed Tradeoff - You're kidding yourself, and also by Uncle Bob responding to Ron's post.  In my current context I wanted to focus on improving the internal quality of the application in a significant way in the not-too-distant future (referring to Joe’s post, I wanted to invest in crossing over the Speed/Quality barrier).  To accomplish this I wanted to dedicate 1-2 developers full time to working on specific tasks with the goal of improving the level of quality within the application (in my next blog post I’ll talk about the specific approach we’re planning on taking to attack this problem).  My first challenge of course was trying to justify taking the developers off of the typical work of implementing new features and fixing defects.  I’m curious how other people in this position justify it?  Do you do as Ron does and try to get your developers to focus on technical debt and trying to do tiny improvements every time they touch an area of code?  What if you want to drive up the quality level faster than that allows?  How do you justify pulling developers off of the work that delivers clearly visible customer-value to focus on internal quality which doesn’t deliver immediate customer-value (at least not in a highly visible way like new features or defect resolution).

When I was trying to justify the cost of dedicating 1-2 developers to focus on improving quality I identified 3 key costs of poor internal quality (or benefits that we could realize by improving the quality):

1. Fragile Code Base - Changes in one area often break things in one or more other areas unexpectedly. This results in either many regressions, or an excessively long QA cycle to prevent regressions. It can also cause programmers to “fear” making some changes due to the unknown impact to the system as a whole, which tends to result in changes that are made in such a way to isolate them from the rest of the code as much as possible (i.e “hacks”), which just serves to further deteriorate the overall application quality.
2. Slow Velocity - Any changes to the system, both enhancements and bug fixes take much longer to implement than would be the case with a system with a strong design. Part of this is due to the fragile nature of the system as described above. More importantly though, the amount of time spent just trying to understand the code will drastically increase with weak system designs.
3. Performance Issues - Another common trend I’ve seen in systems with weak designs is that widespread performance issues are very common. A system with a strong design will tend to centralize behavior and logic, ensuring it’s only run when necessary. Whereas, in systems with weak designs often the same logic and behavior is not only duplicated but executed many different times, possibly in slightly different ways, since one part of the code doesn’t know what the rest is doing. These performance issues can become extremely difficult to resolve since the root cause often requires massive system refactoring to properly address the issue(s).

In my specific scenario I then provided some anecdotal evidence illustrating some examples where we were witnessing these effects.

Anybody else out there have any thoughts about how to deal with applications with weak internal quality?  And depending on the answer to that question, thoughts on how to justify the costs associated with improving the quality?

Traditional estimation involves breaking down the work into tasks, then assigning an estimate of effort in hours/days/etc to each task.  This is then combined into a gantt chart or something similar to create a schedule of the work to be done.

Agile estimation doesn't do estimates at the task level, but rather focuses on the throughput of the team over a fixed period of time.  If the team uses an iterative approach to development the iteration length is a natural time-box to use to measure throughput/velocity, but even if a non-iterative approach is used an arbitrary time-box works just as well (eg. 1 month).  The work to be done also needs to be able to be broken down into "units" of work.  You will want to use something that is easy to break it down into with a minimum of effort/analysis.  Some common units of work are features, bugs, use cases, user stories, etc.  Something such as classes, lines of code, # of methods are typically poor choices as they require a non-trivial amount of effort/analysis to come up with up-front, and even then are typically wildly inaccurate.

Once you've decided on the time period to use, and the unit of work metric to use the only estimating you need to do is to estimate the "velocity" of your team in units of work / time period.  For example, if the team is responsible for fixing defects you would estimate in terms of how many defects per month you can resolve.  Once you have been doing this for a number of months you can simply use the historical velocities from previous months to figure out the average velocity and std deviation to allow you to make an estimate along the lines of "there is a 90% chance we can resolve 25+ defects in the next month".  Or you can make an estimate something like "there are currently 100 defects in the backlog, at our estimated velocity there is a 90% chance we can finish all 100 in 4 months time".  Another common scenario is if you have a target deadline, say a major release in 2 months, you are able to make a statement to the effect "in the 2 months left before the release we estimate - with 90% confidence - that we can resolve 50 of the 100 defects in the backlog, we'll need the stakeholders to prioritize the defects and identify the 50 they wish to make it into the next release".

A common reaction is to think that some defects/features take more time than others, how can such an broad estimation possibly be accurate enough?  But in practice if you are calculating velocity on a batch of work units (eg. The amount of defects resolved by a 3 person team over a month) then the variability tends to average out (the law of large numbers), with the especially long work units being offset by the shorter work units, and if you were to measure estimated velocity (from historical data) against actual velocity you'd find that it is much more accurate than if you were to measure estimated time per task against actual time per task (I believe that studies like this have been done, if anybody has a link to something like that please leave a comment).

There are lots of other benefits to this approach, one is that it provides a concrete source of feedback/input to assess schedule impact - being the velocity measurement at the end of every time period.  For instance, if you have 100 defects in the backlog and have planned on resolving them all over 4 months (based on historical/estimated velocity), at the end of the first month you will have a new velocity metric that you can use to adjust your plan if necessary.  If the measured velocity at the end of the first month is only 10 defects/month then as a PM you will have to take some action to either adjust the plan out to 10 months instead of 4, or do some investigation as to why the velocity has dropped off so drastically and remove some of the impediments to the team to allow them to regain their historical 25 defects/month velocity (and then adjust the schedule as necessary).  This turns out to be a much more reliable process for updating the schedule/plan than relying on developers to provide feedback when they realize an estimate won't be met.  Developers tend to be overly optimistic, and even if they realize that they are 4 days into a 6 day task and not even halfway done yet, rather than having to tell the PM to adjust the estimate, they tend to try to work harder/longer/smarter to squeeze the rest of the work into the remaining 2 days.  This typically results in them waiting until they are almost at the 6 day mark before finally "admitting defeat" and informing the project manager that the estimate won't be met.  More importantly is the fact that the schedule estimates can only really be updated by the developers once they actually start work on the tasks, so after the first month of work you only have estimate updates on 25% (or less) of the tasks, and usually the other 75% of the estimates are left in tact since they haven't been started yet (a strong PM may notice a trend and extrapolate that trend out to adjust the estimates on the other 75% of the work, but in my experience that doesn't happen too often).  Whereas with the velocity approach, it's very intuitive that if you notice your velocity is only 10 vs the 25 estimated this month it makes sense to expect that it will remain at 10 for the upcoming months (unless something is done to affect it, eg. Add more resources, remove distractions, etc).  Then the replanning effort is a simple division of the # of work units divided by the expected velocity, and voila you have your expected project length.

In my experience that is really the information that is needed to perform planning activities, and estimating at the individual task level is usually inaccurate, time-consuming, frustrating, and usually totally unnecessary.

I've been thinking about what makes one chunk of code "better" than another. And in general what qualities do I look for and focus on when writing code.

Personally I believe that the 2 most important qualities of good code (in order of priority) are:

1. Meets customer requirements
2. Maintainability

Now those are pretty broad qualities, but I think I can break it down a little further. How do I focus on meeting customer requirements? Well I believe there are 2 major facets to this, in order of priority:

1. Verifiable Requirements - This usually means executable requirements in the form of acceptance tests. This provides us with a verifiable way to determine if our implementation meets the requirements as stated, whether they be functional or non-functional requirements. However, it doesn't address the issue of inaccurate requirements, thats the purpose of the next area of focus.
2. Short Feedback Cycle - When implementing an application there should be a strong focus on shortening the feedback cycle. This means frequent releases, and high-quality releases. This enables you to identify inaccurate requirements and complements the Verifiable Requirements focus. A large part of accomplishing this lies in maintaining a high-level of quality such that it is always as close as possible to release-ready. This typically involves techniques such as continuous integration, aggressive refactoring, and comprehensive unit tests.

How do I define maintainability? I see it as the combination of 2 sub-qualities, in order of priority:

1. Testability - I believe that testability is the biggest factor in determining whether code is maintainable. Whether you have an existing test suite or not is a separate issue, but when evaluating the implementation code itself testability is key. This means using principles such as separation of concerns, dependency injection, the law of Demeter, loose coupling, etc.
2. Simplicity - After testability the next most important factor in determining maintainability is simplicity. Assuming you have multiple choices for how to implement a feature, and they are all equally testable, the simplest solution wins.

Obviously just stating that the simplest solution is best is a little vague. So lets break down what I mean when I say "simple". I think that simplicity is determined by 3 factors, in order of priority:

1. Separation of Concerns - Each piece of code should have one and only one well-defined responsibility. This allows you to easily isolate pieces of logic for testing. It also helps lead to a set of classes that has high-cohesion and loose-coupling, an important set of design principles.
2. Minimize Duplication - Also commonly called DRY (Don't Repeat Yourself), this demands that you employ aggressive refactoring to drive out duplication in your implementation. Duplication occurs in many forms, and spotting all forms of duplication is a skill acquired with experience. Nonetheless, minimizing duplication will result in a simpler solution that is ultimately more maintainable.
3. Minimize # of classes/methods - Assuming you've first focused on separating the concerns and minimizing duplication, the last factor you should consider is minimizing the # of classes/methods while still retaining the previous 2 qualities. This also falls under the YAGNI (You Ain't Gonna Need It) principle, since if classes/methods don't help to separate concerns or reduce duplication, and the current customer requirements can be met with less classes/methods, then you should use the solution with the fewest moving parts.

So to summarize, I believe that the hierarchy of qualities that are important for code (in order of priority) is:

• Meets Customer Requirements
• Verifiable Requirements
• Short Feedback Cycle
• Maintainability
• Testability
• Simplicity
• Separation of Concerns
• Minimize Duplication
• Minimize # of classes/methods

How would you define "good" code?

I've recently been involved with a project that involves a BizTalk 2006 application.  I've been focusing most of my time on improving the testing infrastructure, and trying to automate alot of the deployment and configuration tasks with the goal of getting to the point where we can setup a Continuous Integration environment.

I'm happy to say that I think we're pretty much there, but not without having to overcome some hurdles along the way.  This is my first encounter with BizTalk and it was a great learning experience.  What I realized early on is that BizTalk doesn't appear to be very tester-friendly.  I'm a TDD guy and I can see that trying to test-drive a BizTalk orchestration could prove tricky. It just seems like it behaves like too much of a black-box, and trying to develop in small increments and isolate specific functionality in the BizTalk components doesn't appear to be easily possible.  I find that testability comes from being able to have control over the component interactions, and the ability to monitor what is taking place.  When dealing with code projects this is typically achieved through loose-coupling, dependency injection, and/or mock objects.  With BizTalk your options are somewhat more limited.

I did manage to stumble onto a project called BizUnit which tries to tackle that problem.  I was brought into this project after the bulk of the BizTalk functionality was already implemented (although without any tests) so I haven't had the opportunity to try it out yet.  If anybody out there has experiences with it I'd love to hear what you think.  Or just how you tackle unit testing BizTalk in general.

Aside from being able to unit-test and TDD the BizTalk implementation, the other challenge has been trying to get a good testing infrastructure setup that is automated, fast, and easy to run.  The idea is that once we have everything setup, we should be able to setup a CI environment that can compile and run the test suite automatically on check-in's.

The main issue has been around the deployment of the BizTalk components.  In order to run any tests against the BizTalk projects they first need to be deployed into BizTalk (and the old versions undeployed).  Deployment is (or was) a manual process involving stopping the old application in BizTalk Admin, deleting the old application, deploying the new version from VS, importing the bindings in BizTalk Admin, and starting the application in BizTalk admin.  And this all had to take place any time any changes were made to the BizTalk projects.

My first thought was that surely all of this could be done from the command-line, all I need to do is throw together a few commands into a batch file or msbuild file to automate this process.  Unfortunately it's not that simple.  About the only thing from that process that can be done from the command-line is importing the bindings file.

Luckily I found the Enterprise Solutions Build Framework from Microsoft Consulting Services that promised to make life easier for me.  It is a set of MSBuild tasks that includes tasks for stopping and starting BizTalk applications, deleting and creating BizTalk applications, etc.  Exactly what I was looking for.Turns out the GotDotNet site hasn't been updated in over 6 months, and most of the links - including the downloads link - are broken.  I managed to find another site that has a sample project available for download that includes the SBF binaries.  It also includes a great sample of using the BizTalk tasks to uninstall and redeploy a BizTalk solution.  I modified the sample for my own use, the only really significant change was replacing his bindings stuff with an exec task that called out to BTSTask to import the bindings (the SBF includes a task to import the bindings, but neither the author of that sample nor myself could figure out how to get it to work).

Here is the MSBuild script that we are using to deploy our BizTalk components (client-identifying information replaced with *'s).  I have a bit of code that runs this in the ClassInitialize of my test fixture.

I believe in having 3 layers of testing: Unit Tests, Acceptance Tests, and Exploratory Testing.  Each of these layers is somewhat independent of the other and each layer alone essentially attempts to validate that the entire system works.  However, none of these layers alone is perfect, but having all 3 in place simultaneously makes the situation much better.

Unit Tests
Unit tests are the tests that the programmer writes as he develops the software.  We develop using TDD and the unit tests are automatically generated as part of that process.  Unit Tests should focus on small pieces of functionality, isolating it from other related functionality.  Typically I accomplish this using mock objects to isolate classes or methods from it's dependencies by mocking the dependencies.  If TDD is performed effectively your Unit Tests should automatically provide a very high level of code coverage (90%+ as a rule of thumb).

Ex. I have an UpdateCustomer method I need to implement.  I will have a Unit Test for the "happy case", I will have a unit test passing in a blank customer name (invalid data), I will have a unit test passing in a duplicate customer name (invalid data), and unit tests for any other validation tests that I need to "test-drive" into the implementation.  I will mock out the DatabaseLayer, and any other dependencies the UpdateCustomer method may use to perform validation or other logic.

Acceptance Tests
Acceptance tests are higher level tests that are written from a customers perspective.  They are not concerned with isolating small pieces of functionality like unit tests but instead test the whole system from end-to-end.  If you ever develop "user stories" as part of your requirements (eg. XP) or a scripted test to be manually executed by a human, then an acceptance test is basically an automated version of those.

Eg. For implementing a customer maintenance application I may have an acceptance test that creates a new customer, updates a field on it, retrieves the customer and validates that the properties match what I expect, then does a delete and another retrieval to validate that the delete was successful.

Exploratory Testing
This is testing that is done manually by a human.  This is probably what most people are used to.  It does not use any test script, it is just simply a person using the system and trying to discover new bugs.  In my experience a human is usually very good at identifying weak portions of the system, and then reasoning where other weaknesses may exist based on what they've seen.  I use this in addition to Unit Testing and Acceptance Testing to help us find any bugs the automated tests may have missed.

Now that I've explained at a basic level what each of the test types I use are, lets take a look at how I use these 3 types of testing in concert.

Typically when I'm developing an application I'll start by designing what I think the UI should look like and how it should function (we currently do not write any automated tests directly against the UI layer).  This ensures I'm focusing first and foremost on the user experience and meeting the users needs; since the UI is the only portion of the application the user sees or really cares about.  I'll draw up the UI and write up the code that interacts with my business layer (typically a web service).  In the process of doing this I discover (aka design) how I want/expect the interface on the middle-tier to look like to make it easy for the UI to consume.

Obviously the UI won't compile yet, as the Web Methods I have been coding against and designing as I go don't exist yet in the middle-tier.  So the next step is to create skeleton methods in the middle-tier (Throw New NotImplementedException); just the bare minimum of code needed to get everything to compile.  At this point I'll usually check my code into source control since it is at a compilable state and no failing tests (since I haven't written any new tests yet - that's next).

My next step is to write the first Acceptance test.  This will exercise the web methods from the middle-tier that we consumed while coding the UI.  Once I get a reasonable Acceptance test written up, it should compile, and fail (since all the web methods should be throwing NotImplementedExceptions currently).  This Acceptance test represents the first "user story" that I'm going to focus on implementing.  Now that I have a goal (getting the Acceptance test to pass) I'll start implementing the middle-tier methods using TDD to write unit tests and implementations as I go.

I'll keep using TDD to drive out the implementation necessary to get the Acceptance test to pass.  Eventually I'll get to the point where I have enough implemented that my Acceptance test will pass.  At this point I should have the one passing Acceptance test and quite a few passing unit tests as a result of the TDD.  At this point I'll check in my code into source control (good time to do it since all tests are passing).

The next step is to write another Acceptance Test exercising some more functionality that the first Acceptance Test didn't cover.  Then I'll TDD the implementation required to get this Acceptance test to pass.  Then check-in to source control.  This process repeats until I'm satisfied that I have enough Acceptance tests to cover the functionality required. 

When I use TDD to drive the implementation I stick to the rule that I'm only implementing strictly what is required to make the test pass.  If I feel there's more logic needed in the implementation, I won't write it until I first have a unit test to validate it.  However, when I'm using TDD to drive the implementation of a specific Acceptance test, I do not stick strictly to what I need to make the Acceptance test pass.  For instance, lets say I have an Acceptance Test that calls UpdateCustomer() web method.  While I'm TDD'ing the implementation of this UpdateCustomer web method I'll create unit tests to drive out all the functionality I believe is required in that method.  For example, I may have a handful of unit tests that just call UpdateCustomer() passing in various different types of invalid data.  This will drive the implementation of the validation logic in UpdateCustomer.  I  write these unit tests and implementation, even if that logic isn't specifically exercised by the Acceptance Test I'm currently trying to implement.

Once I have all the Acceptance Tests I feel necessary written and passing then I should be done implementing everything for this portion of the UI as far as I'm concerned.  Now it's time for some exploratory testing.  I will typically perform at least a little bit of exploratory testing myself just to ensure that the UI works properly and everything I needed to implement is in fact completed.  Then you should have somebody with fresh eyes do some more exploratory testing (ideally a dedicated tester in conjunction with the end users).

Anytime a bug is found while performing exploratory testing this indicates that one or more automated tests are missing from the test suite.  There should be at least one unit test to demonstrate the bug, and depending on the nature of the bug you may wish to write an acceptance test in addition to the unit test to demonstrate the bug.

Another thing I would note, is that if anytime during the implementation or maintenance of the application you wind up with a failing acceptance test (that was previously passing) but no failing unit tests, the first step should be to write a failing unit test, make the unit test pass, then check to see if the acceptance test is now passing.  If not write another unit test and repeat.

I feel that having these three layers of testing provides the safety net that enables pain-free refactoring, along with a strong focus on quality code.  Unit Tests take care of testing the basic pieces of logic in isolation, Acceptance Tests ensure that all that logic integrates properly and does in fact work together as a complete system.  Exploratory Testing helps to capture all the bugs and logic that the developer forgot about while writing the unit and acceptane tests.

Let me know what type of testing strategy you follow?  Do you think this sounds like too much effort?  If so what would you do differently?

I was chatting with a buddy of mine who works in consulting about the Project Status Report I've been using (see previous entry).  He was concerned that something like that wouldn't work for a consultant (both the report and the process itself) without some modification.  His main concern was that they need to track billable hours for both project costing and billing the client.  My process and report don't deal with hours instead choosing to deal with "work points".  Work Points represent the amount of effort required relative to the other features on the list.  If you wanted to I'm sure you could spend some time and track hours worked and then calculate out how many hours worked correspond to each work point.  In my circumstances though that is not needed.  Work Points gets me an abstract velocity number which allows me to easily detect if we are speeding up or slowing down our progress, and allows me to calculate an estimated completion date which is really all I'm interested in.

For a consulting company, they are interested in hours for calculating costs and billing clients.  We were talking about how this report and process could be adapted to work in his situation.  I thought about it for a while and have an idea that is probably pretty out there (but hey, these are the things I like to think about).

Instead of charging clients by the hour why not charge by the work point.  It would probably be more attractive for the clients (once they got used to it) since now they are paying for value delivered rather than hours worked.  And the cost won't fluctuate depending on how many hours it takes (although I'm guessing consultants provide hours estimates up front that the clients are charged for regardless of the actual hours).  I can see an advantage to consultants also, but let me rehash a conversation I had with another consulting buddy a while ago.

I was talking with Joel Semeniuk about his consulting company (Imaginets) we were talking about a situation he encounters there. Imaginets has extremely low employee turnover.  I take this to be a sign that they know what they are doing and are a great company to work for.  However, the problem this presents is that their employees are getting more and more senior as they gain experience and demanding higher and higher salaries.  What is the company to do?  Do they start charging clients higher and higher rates?  That may be one option, and you could certainly argue that it's justified.  Joel has some novel ideas on how to deal with that situation, but lets get back to the original conversation.  If you were to charge clients based on work points instead of hours worked it seems like it would help in this situation.  The client and the consultants don't need to deal with how to charge for senior devs vs junior devs and all those inbetween.  If they just charge based on work points it's up to the consultants how to staff the project.  If they put junior devs on the project the assumption is that they would have a lower velocity than a senior dev but it would even out because their salary is presumably lower to compensate.  Likewise, paying the senior devs more would be justified assuming that their velocity is high enough to justify it.  All the consulting company would need to worry about is that the aggregate velocity is sufficient to hit the date given to the client.

It also removes the task of having to provide up front estimates in hours of how long everything will take.  Instead the estimates can be done in abstract work points which in my experience are infinitely easier to estimate.

Now my knowledge of consulting companies and their processes and issues is somewhat limited, so there's a good chance this is a crazy idea.  If so give me some feedback and let me know the many reasons why it won't work.  Or maybe it would work, maybe lots of consulting companies already do something similar and I'm just not aware of it (easily possible).

I wanted to take a few minutes to talk about the project management / tracking / estimation methods I use.  Over time the processes and tools I use have changed quite a bit, but the way I've been doing things recently really works well for me.  The primary tool I use to manage a project is a feature list.  Similar to a product backlog from SCRUM (although we don't do sprints...or I suppose you could say we do a sprint for every feature).  The feature list is constantly evolving as we discover new requirements, or new feature requests are received.  I maintain this list using a Sharepoint list on our project site.  The feature list is prioritized (stack-ranked) in order from highest priority to lowest.  We don't give each feature a priority from 1 to 10 or some similar ranking (low, medium, high, etc), instead we sort the whole list according to priority relative to the other items on the list.  This prioritization is done as a group with the project stakeholders, and we hold a weekly meeting where any new feature requests are reviewed and their priority relative to the rest of the list determined.  One of the nice things about using this method, is that we never have to refuse a feature request, or somebody requesting to expand the scope.  Every request will be added to the list, then it's up to the group to decide where it should be prioritized.  Even feature requests that are just wish-list type stuff, that we know won't get done in the near future can still be added to the list, they will just be prioritized near the bottom.

At our weekly meeting, in addition to reviewing and prioritizing new feature requests, we also review and update (if necessary) the "go live" point.  This is the cutoff point in the feature list that all the stakeholders feel includes enough functionality and benefits that we can roll out the application into production.

Each item in the feature list is also assigned an Effort Required rating from 1 (easy) to 5 (hard).  This is just a rough estimate done by me (the development manager) that gives everybody an idea of the work required to implement each feature relative to the other features.  If I find myself giving a rating of 5 (hard) I will usually attempt to break the feature up into 2 or more smaller features that can be tackled separately.  If I feel a feature is a 1 (easy) I will see if I can group it together with another related feature.

Along with each feature we also maintain a % Complete which is updated by the developer(s) working on that feature.  We define specific percentages to represent key points (milestones) in that features lifecycle.

0-49% is under development
50% is ready for review by the feature owner(s)
60% means it has been reviewed
61-79% means it is undergoing revisions as a result of the review
80% is ready for demo to the larger group (at our weekly meeting)
90% means it has been demo'd
91-99% is undergoing final revisions as a result of the demo
100% is completed

Now that we have the effort required rating for each feature (I also call these "Work Points") and the % complete, we can do a number of useful calculations.  We can multiply out the % complete by the Work Points for each feature and sum it up to get the total work points completed.  If we compare this to the total work points required we can come up with a % complete for the whole project.  We can also track the # of work points completed each week/month to determine the velocity.  Using this we can calculate the estimated time required to complete all the remaining work points.  I produce a status report every week that includes all these calculations.  Here is a sample of what it looks like:

The beauty of this report is it is extremely easy to generate (when I get some spare time I may write some code that automates the generation of this report by reading from sharepoint directly).  It also provides very useful information, to me, the project stakeholders and upper management.  The % Complete number gives a quick update on the progress being made on the project.  And the Estimated Go Live Date is a useful piece of information to compare against any previously created timelines or deadlines.  Since it's based off of real measurements of progress it makes it much more useful than somebodies (usually mine) wild estimate of when we'll be done.

Do any of you guys do something similar?  Let me know what you think in the comments.

I will be giving a presentation on TDD to the Winnipeg .Net Users Group on Jan 9.  Should be an interesting talk.  Got about 2 hours and I want to go through a quick powerpoint and talk about some of the core concepts and benefits.  Then dive right into developing something simple with TDD.  Not sure what I'm going to use yet for my coding sample.  I might take the classical bowling game example.  But Bowling has been done ad nauseaum...  Sudoku could be fun to do, but last time I tried TDD'ing that I ended up using mock objects and I was hoping to avoid mocks and interaction based testing to keep things simple.  I could also do some boring business-type scenario, transferring funds from one bank account to another or something equally boring.  I think some people might prefer an example like that as it is more applicable to their day-to-day work; but to do any business-type scenario it almost has to include a database which means using mock objects in the tests.  I'll just have to do a test-run of a couple of the ideas and see which one feels best.

Jean-Paul Boodhoo was telling me that he likes to just let the audience pick out a problem.  That sounds like a great idea, can help show how easy it is to apply TDD to any problem by not having a pre-canned demo.  However, then I wouldn't be able to steer clear of some of the more advanced concepts like mocking and inversion of control principles.  If I had 1-2 more hours then I'd just cover mocking also and could tackle pretty much any type of problem as a demo...but in the 2 or so hours I have I just think it would be too rushed.

Anyways, it should be an interesting talk I hope.  Anybody out there who is interested in TDD or in techniques that produce clean, reliable, and easily maintainable code should try and make it out.  Also if anybody has any input as to what they think I should cover (or not cover) or use as a demo problem feel free to contact me (optikal "at" shaw.ca).

In a previous post I mentioned that I had been trying to use Team Build to get a Continuous Integration environment setup.  I recently had the time to return to that task and have managed to get most of what I wanted up and running.  I had to jump through a few hoops to get everything working so I'm going to try and describe what I had to do to make everything work right.

My solution consists of a WinForms UI, ASP.Net Web Service, 2 Class Libraries, Database project (VSTS DBPro CTP7), Testing project, Setup project for UI, and a Setup project for web service.

My testing project contains just over 250 tests.  About 150 of these are what I call "unit tests" that directly invoke methods in one of the DLL's and mock out the database (they run very fast; about 5 secs for all 150).  The other 100 are what I call "acceptance tests" that operate at a higher level, and invoke calls against the web service (I have no tests against the UI project, or directly against the database at this point).  The acceptance tests require the web server to be up and running, and require the database to be wiped out and re-initialized on every test.  It takes about 20-30 mins to run the full suite of 150 acceptance tests.

The first thing I did was create a Team Build using the built-in wizard.  I didn't specify any tests to be run initially.  I had to do a little tinkering around to get the Team Build to work.  There were a few dependencies that I needed to either add to our source control, or install on the build machine.  I had some problems with the setup projects in there, and the database project (the problems with the database project were back in DataDude CTP3 I believe they are fixed in the latest CTP).  What I did to work around these issues was just create a 2nd solution that only contained the WinForms, Class Libraries, and Testing projects.  I had some custom code in the testing project that would auto-deploy the database with the tests, but I had to make sure the build output (the .sql file) from the datbase project was included in source control. 

I did notice that the Team Build didn't seem to like to build my Web Service project.  After a bit of research I found this post by Joel Semeniuk about this issue and how to resolve it.  I had to modify a section of the TfsBuild.proj file to look like this:

<ConfigurationToBuild Include="Debug|Mixed Platforms">
  <FlavorToBuild>Debug</FlavorToBuild>
  <PlatformToBuild>Mixed Platforms</PlatformToBuild>
</ConfigurationToBuild>

Not a big deal once I knew how to do it.  Now the Team Build ran and correctly built all 4 of my projects.

At this point I had a Team Build that I could run on demand from within VS and it appeared to work great.  Now I just had to get it to run my tests as part of the build, and to run automatically on every check-in.

I knew from before that there would be issues with trying to use and maintain Test Lists in our environment since we mostly just use VSTS Dev.  This post by Buck Hodges gave me hope that there was a nice workaround available.  First things first though, I just wanted to get the Team Build running with my tests.  The easiest way to do this was to just fire up Team Suite and create a Test List to use temporarily.  I created a Test List and added all my tests to it.  Next step was to modify my Team Build to run the tests as part of the build.  In my TfsBuild.proj file I had to update the <RunTest> element to True, and add the following section in there:

<MetaDataFile Include="$(SolutionRoot)\ShopFloorTeamBuild\ShopFloorTeamBuild.vsmdi">
  <TestList>BVTList</TestList>
</MetaDataFile>

This tells Team Build to run the Test List BVTList as part of the build.  I fired up my build and noticed that the Unit Tests were all working correctly, but all my acceptance tests were failing.  I had expected this to happen, as there is no Web Server running on the build machine for the tests to hit against.  To fix this I had to add the <AspNetDevelopmentServer> attribute to all of my acceptance tests and add a line of code that performs a little magic on the client-side to redirect the invocation to the proper address.  More info on how this works can be found here.  This is how my code looks for one of my acceptance tests:

<TestMethod()> _
<AspNetDevelopmentServer("MyWebService", "%PathToWebRoot%\MyWebService")> _
Public Sub MyAcceptanceTest()
    Dim Target As MyWebService = New MyWebService
    WebServiceHelper.TryUrlRedirection(Target, Me.TestContext, "MyWebService")
    ...
End Sub

I ran into a little trouble getting this working correctly.  It seemed that I could get it to work either on my local machine, or on the build server, but not both.  It seemed that they were interpreting the %PathToWebRoot% variable differently.  After a little more research I discovered that I can set it up so that it works on the build server, then for the local machine there is a local setting I can change to override where %PathToWebRoot% will point to locally.  It can be overridden in Tools->Options->Test Tools->Test Execution, the Web Application root directory property.

At this point my Team Build is building on demand, and properly executing all of my tests.  I'm getting close now, I can almost taste it.  Time to try and make it run without the test list.

My first attempt was to follow the directions given by Buck Hodges in this post.  I downloaded the task and targets file and set them up on my build server.  Then I modified my TfsBuild.proj file like so:

<!--
<MetaDataFile Include="$(SolutionRoot)\ShopFloorTeamBuild\ShopFloorTeamBuild.vsmdi">
    <TestList>BVTList</TestList>-->
</MetaDataFile>
-->
<TestContainerInOutput Include="ShopFloorTests.dll" />

I tried running the build and all the tests run, however all my acceptance tests are now failing.  After a little investigation I discover that a config file that all the acceptance tests rely on is not getting copied to the tests directory when the tests are run.  I had previously used the .testrunconfig file to specify that this file should be deployed along with the tests.  It appeared that the <TestContainerInOutput> wasn't using my .testrunconfig file (this was confirmed in the comments from Buck's Post).  I was doing some fiddling around with the TfsBuild.proj file, and I had commented out the <TestContainerInOutput> tag, and uncommented the <MetaDataFile> tag but deleted the <TestList> part of it.  It appeared that it was still running all my tests, even though I wasn't specifying what test list to use.  So now my TfsBuild.proj looks like this:

<MetaDataFile Include="$(SolutionRoot)\ShopFloorTeamBuild\ShopFloorTeamBuild.vsmdi">
</MetaDataFile>
<!--<TestContainerInOutput Include="ShopFloorTests.dll" />-->

All the tests run and pass.  I tried removing a test from the project (not touching the vsmdi file) and re-running the build, and sure enough it didn't try to run the deleted test.  I also tried adding a new test, and the build picked that up automatically also (without me needing to modify any test lists).  I'm not sure what I did, but I could swear it never used to work like this.  Maybe it's a result of installing Buck's new task/targets files.

The next step is to get this build running automatically on every check-in.  I followed the approach used by Jeff Atwood here.  That turned out to be pretty straightforward and easy.  Now I just need to setup something so that we can be notified when the build is complete.  After a little reading I discovered the Alerts system built into team foundation server (I've never really noticed it before).  All I needed to do was go Team->Project Alerts...  check off the "A build completes" alert and type in my email.  Every time the team build completes, I now get an email with a link to a build results webpage.

So far so good.  I have my team build running automatically on every check-in.  It compiles all my code and runs all my tests.  It sends me a notification email when it's complete.  That's all the important stuff working.  I still have to play around with getting Code Coverage working in the automated build.  And I would really like to get a build report that just shows me the differences between that build and the previous build as I talked about here.  The most important features of a CI system though are all up and running.

I know the TFS Build team at MS is working hard on the next version, and hopes to make alot of improvements.  I'm hoping that this process will be a little more straightforward in the next version of TFS, and that there will be some added features (such as diff reports between builds, and CI out of the box) that enable us to be more productive and effective at what we do.