Using Agile Project Management with Integration Projects
I have been recently involved in a number of BizTalk 2004 integration projects using Agile Project Management (Also known as Scrum). Conchango (http://www.Conchango.com) has also been involved with Microsoft in designing and implementing the Agile add-in for VS2005.
If you are unfamiliar with Agile Project Management, have a look at the following links:
http://www.controlchaos.com/ (Its all about common sense) The official Website with all the resources
http://www.scrumalliance.org/ For Certified Scrum Master Resources
http://wiki.scrums.org/index.cgi?HomePage and http://c2.com/cgi/wiki?ScrumProcess
Background
First, a bit of background, I don’t want to assume that you are an expert on the subject and will try to give a bit of info to get you going for this post.
Agile project management was introduced for development projects. The standard sprint duration is 30 days and at the end of each sprint the project team presents “Production Quality” features. At the beginning of the project the team organise their deliverable (Called Products) in a “Product Backlog”. A representative (Called Product Owner) from the stake holders helps them identify the products and prioritise the product backlog. The team starts with the most valuable tasks first, then the less valuable till they deliver the entire system or run out of budget. The emphasis is on two things: The stake holders are seeing results very early on. So after four working weeks the team would have produced real business value. The second is that the products of each sprint are production ready. So, if the stake holders decide to put this to live today, they can do that.
The Chickens and the Pigs
The first time I was called a chicken I was insulted! Then I was called a pig and I was insulted again, well not really because by then I knew what it meant! The story is that a pig and a chicken wanted to open a restaurant. The pig asked: What should we call our restaurant, and the chicken replied: Let’s call it Ham and Eggs. The pig thought about it for a minute and said, I can’t have a restaurant with you because I would be committed and you would only be involved. Therefore, only the scrum master, product owner, and members of the team are pigs (Committed), all the rest are chickens (Only involved). The advantages of this are countless: How many times were projects ruined/delayed, went over budget and/or lost focus because of restrictions imposed by non relevant people. How many times did meetings drag on because you had to involve everyone in the department? How many times where you misinformed because you could not find the correct answer or talk to the right person. Being a pig (If you excuse the expression) not only empowers you, it also clears unnecessary obstacles, and gives you direct access to other pigs.
The scrum meeting
The scrum meeting is the hallmark of Agile project management. It is a daily meeting, capped at 15 minutes, held in the same place, only pigs are allowed to talk, chickens can listen. Each person (Pig) tells the team what he/she has done yesterday, what he/she are intending to do today and finally mention any impediments they have.
Tip1: Have the scrum meeting standing up and away from your desks. This ensures people focus on the meeting, and prevents the quick peak to the old email inbox
Tip2: Pigs stand in a circle, chicken outside the circle. This enforces the separation between committed and non committed resources
Tip3: Establish a fine for those who are late for the scrum meeting. Mine is a pack of Bahlsen Choco Leibniz dark chocolate biscuits (http://www.bahlsen.co.uk/), or two if you someone has been really naughty!! Most people though tend to gather the money and later donate it to a charity of their choice.
Tip4: The daily scrum time is very important. If you make it too early, you might have people stuck in traffic and start having problems with people missing very often. Having your scrum meeting too late, say at 10 am or 10:30 am, people will tend to do whatever (not work related) waiting for the scrum to begin. Scrum time of 9:30 am is on average a good time.
Semi-committed pigs*
Sometimes you require resources for a short time only, or have to share them with other projects. I call these Semi committed pigs. The first problem with having semi committed pigs is time management and estimation. They will often say: “I have to work on this other project, so you can have 40% of my time.” This is very difficult to estimate because they will not always give 40% every day. The second problem is that you always try to have the team in the same room. Semi committed pigs will generally not be able to do that because of their other engagements. This introduces plenty communication issues.
* The term “Semi-committed pigs” is truly my term and you will probably not find it in any of the books. If you do find an equivalent please let me know.
Why use Agile for integration projects?
So far I have been involved in four (and counting) BizTalk project using Agile. I am a certified Scrum Master. I had the privilege of working on the first integration scrum project ever. The following is why I think Agile is very well suited for Integration and BizTalk Projects.
Communication
Having daily scrums with relevant involved stakeholders is the biggest advantage Agile projects have over other Traditional approaches. In addition, there is a benefit of delivering functionality early to business users:
- The feedback is much more relevant to what has been delivered so far. Therefore if the delivered functionality is not completely correct, it could be easily altered as you did not build any additional functionality on top of it, and the development knowledge is still fresh in developers’ minds
- The feedback is immediate, and updates the team’s knowledge of the system. This is very important especially if in the next sprints there will be functionality that either uses this feature or extends it.
- Reprioritising functionality: Very often stakeholders will re-evaluate their priorities because they have identified additional benefits in what has been delivered making other products on the backlog either redundant or less attractive. This is key as, in traditional projects, we see large budgets being spent on features that are rarely used. This is because the business did not get to see the solution until development was completed for all the features of the system.
Project Inception vs. Scrum Sprint 0
In traditional waterfall projects business analysts, technical architects, and project managers spend considerable amount of time identifying the business requirements, business processes (Existing and changing) existing legacy applications to integrate with, human interaction, etc. They try to define this for the entire product they are supposed to deliver at the end of the long term project. Not only is this a very complicated task, but by the time you reach, say, user acceptance, this is often outdated because everyone would have gained much better understanding of what actually needs to be done. Also the complexity of the systems we usually build forces us down routes that were not anticipated at the beginning of the project. The end result is a change request process that usually drags on, as well as out of date documentation. The other problem with the waterfall approach, and I had this recently, the inception process is very loose and undefined by definition. The room for error is very large especially that the waterfall model does not enforce the concept of daily update meetings. The result is that business analysts, Technical architects, PMs would have initial project meetings with the stakeholders, go away for few weeks and think about it, and come back with a design and a project plan that they feel would solve the problem. Doing this, the stakeholders are led down a road that the “Experts” thought through, and they will trust them to have considered all the options. The point is, that there was no daily involvements from the stakeholders on the decisions being made during this very important phase of the project. The end of inception is usually one or very few options that are presented and discussed. In my experience, we started a project using scrum after the inception phase produced an invalid solution. I was lucky because I was critical and questioned everything before I started development. It took some time to discuss the shortfalls of the proposed solution and convince my client to adopt what I called “Sprint 0” or Sprint Zero. Because it was agile, we had daily scrum meetings with a representative from the stakeholders, a product owner. Every morning we discussed our options, and made decisions together because we were on the same level. At the end of sprint 0, we presented our alternatives to the rest of the stakeholders and really felt that by that time we have reached a decision together. Everyone was much more involved just because they were part of the decision making process.
So what should be in Sprint 0?
Basically this should include everything to get the project started with Sprint 1. Therefore, this sprint could include everything from identifying the business case for the solution or project, identifying business benefits, revenue potential, and cost savings, present the different cost options, rollout strategies, etc This sprint could also include decisions regarding vendor selection, pricing models for hosting systems, leasing hardware, evaluating Components Off the Shelf (COTS), identifying the tasks that could potentially be on the product backlog, and the team structure that could be required and their skill sets. In one project we talked about peeling the onion, i.e. identifying the layers of complexity and costs with the knowledge that we still have to be agile and account for changes and modifications in later sprints. At the end of sprint 0 you should end up with a working product backlog, or at least one that will let you start the first sprint. Also consider that you might finish the project without doing all the items of the backlog, as well as being prepared to stop the project after each sprint review and deploy your application to production.
Sprints and Sprint Duration
The standard sprint duration is 30 days. Sprint durations should be equal. I used 10 day sprints for my first agile BizTalk project. The advantage is that it fits with the average time you need to build a demonstrable BizTalk Project, or part of it. In that project I had to write a custom BizTalk adapter that allows us to integrate with Chip and Pin devices. That was easily done “Production quality” in a ten day sprint. On the other side of the interface was integration with the AS400 system using Attunity. Again this was easily done in a ten day sprint. I passed this by Ken Schwaber (The father of Agile) and he was fine with. He said: As long as you are fine with it I’m fine with it! The disadvantage of a ten day sprint was that external vendors usually fund our pace overwhelming and struggled to keep up with our demands. We were too agile for then and I think they were not used to that. The second disadvantage was with Semi committed pigs. The short sprints allowed little room for error and if the semi committed resource is stuck with another non project related task you could not compensate easily.
Architecting SOA the Agile Way
It is now agreed that SOA increase the flexibility and the reusability of software components. As enterprise applications evolved and the adoption of SOA increased, integration among all the services became critical. By definition, each service is an independent component, has its own schema(s), unit tests, architectural views, etc. Architecting SOA the agile way is about incremental delivery of production quality services. This could mean that in certain sprints the development team will deliver production quality incomplete services but they are complete in the context of the sprint. In other words, the service would do the functionality required for this sprint and subsequent sprint will extend that service further. The result is a business targeted delivery rather than an IT based approach. At the end of each sprint, it is the decision of the stakeholders if they would want to move this service to production and reap the rewards of that functionality. This is a very strange concept to developers as they are used to dealing with complete architecture, and technical specifications when they start coding. In fact, if the agile process is implemented, the development team should have an incremental architecture with every sprint, which should be much more accurate and up to date in comparison with what would have been produced in a waterfall planning session. Architecting SOA the Agile way is really about implementing corrective incremental steps towards the end result. In golf talk, in a waterfall context the architect is expected to score a “hole in on” every time, while Agile is all about taking steps in the right direction.
Coping with Change
If I had a pound for every time someone changed his/her mind on a project I would have been a very rich man indeed. Actually, it is more likely that business analysts, stakeholders, project managers, developers, and admit it even architects, will change their minds during the duration of the project. In fact, the longer the project, the more likely change will be introduced. Last year I worked on a large project integrating SAP, Siebel, and AS400 legacy applications. Change was the only thing that hasn’t changed throughout the project. Even during “Code freeze” periods, we where often asked to implement the changes in the middleware interfaces because of the code freeze in SAP and Siebel. Modular delivery does help cope with change, for a start you don’t have to design everything up front and change it later. In addition, you don’t have to spend a lot of time designing everything only to implement part of it, or even none of it. Delivering incrementally will put the team in the right frame of mind required for the change because you only plan for this sprint, and if things change then you will plan for them in the next sprint and so on. Of course if things change for this sprint then the team will have a problem, but I think this is where a shorter sprint duration comes in handy. In integration projects things can change all the time. Even when we were doing the SAP and Siebel interfaces we knew exactly needed to be delivered, but testing was an issue; not everything could be tested the same way or by the same testers. Therefore planning for what takes place when and what to work on next is a challenge. Shorter sprint durations, say ten days, might be an advantage during some integration projects, especially if you want to deliver small modules and reach you goal quicker coping with change along the way. On another project I had to share some resources, and my project was not the most critical. With shorter sprints I could re-organise the sprints to cope with resource availability, a situation not ideal but will be the difference between frustrated demoralised project team, and well, not!
And Finally
These were few points from my personal experience, and every time a new agile project starts and finishes there are loads to learn. If you are still reading, I hope you have found this useful, and I look forward to your comments and questions