Let´s get real about software development: It´s never going to be a quietly flowing river. Never. And that´s why the current approaches to software development like XP, Scrum, and Kanban will always cause pain.

Their basic assumption is you should be able to isolate a team for a while to work on features. Leave it alone during an iteration or a sprint to complete a set of features, or at least sit still until the current feature is done.

Certainly that´s what we all want as developers: being able to concentrate on creating value by implementing new features or at least improving existing ones. Until we´re done.

Project reality seems to differ, though. And it does not just differ because Scrum or Kanban haven´t been implemented faithfully. Project reality will always differ because customers and users don´t care about our longing for being left alone.

Get real about emergencies

Software development is not like some factory production where orders are worked on in a smoothly flowing manner until fulfilled. Software development is more like an emergency unit in a hospital. It´s constantly under fire from all sides.

Requests are coming in at all times. Resources are always limited. That means a software team has to use triage (http://en.wikipedia.org/wiki/Triage) to determine what to do. Three request categories seem appropriate:

1.    Requests that must be answered immediately.
2.    Requests that should be answered soon. If not right away, then tomorrow is still ok, but not much later.
3.    Requests that can be answered at no specific date. Tomorrow would be ok, also next week, maybe even next month.

Category 1 is about rare bugs causing heavy damage with customers right now or in the foreseeable future; show stoppers so to speak. They need to be taken care of within a minutes or hours. Whether certain requests justifiably are categorized as show stoppers or not, is of no importance. Someone high enough up the ladder thinks a request is in category 1, so the team needs to react immediately.

Naturally category 1 requests should be rare. More than once a week seems problematic to me. Not because such interruptions are to be avoided, but because such urgent requests either hint at very poor software quality or an undue relationship with the customer base.

Category 2 is about most bugs. They need to be cared for pretty quickly, but not immediately. Bugs are a threat to any customer relationship. Customers rightly assume bug free software – with bug meaning an unintended deviation from what a software developer was trying to deliver. A bug is the fault of the development team so it needs to take care of it as quickly as possible.

Category 3 is about new features, feature change requests, and corrections due to uncovered misunderstandings or deeper insight. No life is at stake if such a request is not processed immediately. It can be put on a backlog for prioritization.

Any project not just started is under fire with all requests from all three categories. In an early greenfield project, requests might just fall into category 3. But as soon as customers/users have put their hands on some (early) release, category 2 and 1 requests will start to come in.

That´s a fact no software development process should deny or neglect. Right to the contrary any software development process should revolve around this basic fact of software development reality.

My conclusion #1: We need a software development process, which institutionalizes request triage.

Get real about uncertainty

At the heart of Agile and Lean is the realization that requirements are notoriously poor. As much as we invest in requirements elicitation… the result still is fuzzy. I´d even say: Customers cannot specify what they want, they can only recognize if that, what software development releases to them, is what fits their needs.

Customers are poor at a priori definition, but extremely good at a posteriori judgment.

As software developers we thus live in a constant state of uncertainty. We just don´t know if what we´ve developed really is what gets accepted by the customer. Did the customer define his requirements correctly? Are they complete? Are the comprehensive? Does the customer know at all what´s necessary to solve her problem? Have we understood the requirements? Have we come up with a solution matching our understanding? Will the customer accept our view?

Because of this uncertainty Agility/Lean advocates iterative development with delivery of completed features. Whatever get´s released to the customer at the end of an iteration has value to him; he can give feedback, because some requirement has been implemented.

How long until such a release? Two to four weeks seem the norm. Sometimes it´s just one week, sometimes it just takes as long as it takes for a feature. Iterations usually are measured in weeks, not months, not days or hours.

This seems ok – until iteration timeboxes collide with project reality. Any request of category 1 or 2 is likely to cause a delay. Such requests naturally are a nuisance for any process requiring quite time to work on a plan laid out for a couple of weeks.

My conclusion #2: We need a development process, which embraces the undeniable reality of category 1 and 2 requests.

But not only are timeboxes of several weeks in contradiction with daily project life, they also still do not really honor the uncertainty of requirements. Think of it: A day 1 of an iteration you start working on a feature. You finish it by day 4. But the only gives feedback at the end of the iteration, maybe on day 14 or 21. That´s a long time since you checked off the feature in your head as done and got rid of your mental feature state. The feedback loop is quite long even in today´s agile processes. On average it takes at least half an iteration until you hear back from the customer.

Also, during an iteration several features are implemented. This means, whatever change request the customer might have at the end of the iteration, it can possibly affect all features under development within the same iteration. The longer an iteration the more likely waste is created.

My conclusion #3: We need a development process, which allows for almost arbitrarily quick feedback from customers.

And then think of it: If requirements are so uncertain, why should a team deliver whole features at all at the end of a multi-week iteration? Neither does the team know, if it interpreted the requirements correctly, nor is it clear to the customer, if she really needs all that which she found important many weeks or months ago during requirements elicitation.

If a process tries to deliver as much of a feature as possible in a multi-week timebox, there is hardly any chance for the customer to say, “Stop! That´s good enough.” Current processes implement requirements in a very coarse grained manner. Their ideal is completion, arriving at Done for a whole use case or feature. That seems like a relic from the old waterfall days, to me. It´s not very true to the fundamental realization of Agility: Requirements are notoriously uncertain; we need to deliver often.

My conclusion #4: We need a development process, which allows customers to stop development of features at almost any percentage of completion.

Get real about evolvability

Before Agility the biggest challenge for software development was technology. We struggled to fulfill the explicit functional and non-functional requirements of customers.

The next big challenge then was correctness. Also a non-functional requirement, but not often explicitly stated. Customers rightly assume software to be correct, i.e. bug free. There is no excuse for a crash or an incorrect calculation.

Since then, though, the world has moved on. There are tons of technologies and tools at our disposal. There are several systematic approaches to increase correctness of code from reviews, to automatic unit tests, integration tests, and acceptance tests.

So I daresay today´s biggest challenge for software development is different. It´s harder to solve, because to assess the quality of our solutions with regard to it, is more difficult.

What I see as the biggest current challenge is maintainability or evolvability. We need to do a much better job keeping our code flexible. Cost is rising too steeply over the lifetime of a code base. Implementing a feature in month 3 of a project might cost X dollars, implementing it in month 18, though, will cost much, much more.

There are several approaches to attack this problem like Clean Code, Agile Architecture, or just tried and true principles and practices from 60 years of software development.

Strangely, however, current software development processes do not address the challenge. Neither Scrum nor Kanban are interested in code evolvability. They might measure a decrease in productivity, but they have nothing to say about how to improve the situation.

XP might be different in that regard. But then… how effective can it be to have list of principles and practices for high evolvability, when evolvability is not put to the test?

Yes, I think that´s a common blind spot of all current development processes: they do not really put evolvability to the test.

And why should they? Isn´t evolvability a matter of code design instead of process? Indeed it is – but alas there is hardly a measure for the quality of code base evolvability. Forget about LOC per method or class, forget about Cyclomatic Complexity, forget about afferent coupling etc. Those are just numbers that must be interpreted. And this interpretation is highly subjective and relative. Let´s get real: We can´t look at any set of measurements today and positively say, “This code has high evolvability.”

Checking the functional quality of code, checking it´s correctness, checking its non-functional qualities… that´s all different and comparatively easy. Checking evolvability, though, is hard. In the end the proof is only in the pudding. If a code base is hard to evolve, then, well, its evolvability must be low.

To check the fulfillment of functional requirements, execute the program. To check non-functional requirements, execute the program. To check the evolvability, evolve the code.

Sure, design/code developed using XP, Scrum, Kanban is put through evolutions more often than in a waterfall process. But my perception is, that´s not enough. There are so many teams trying to improve evolvability by applying Clean Code principles etc. – but the results are still suboptimal. From that I draw my last conclusion:

My conclusion #5: We need a development process, which addresses today´s major challenge evolvability head on by putting much more pressure on a code base through shortened evolution cycles.

Summary

Current development processes are a big step forward compared to whatever was the norm before. They have done a lot of good to the state-of-the-art of software development. To me the major contributions are:

• We now know that requirements are very uncertain. The problem to solve is always poorly understood by customers and developers alike. To arrive at a solution we need to interact frequently.
• In order to home in on a solution, a development team needs to deliver artifacts to customers on which they can give feedback. Software thus is delivered in slices (vertical sections), not layers (horizontal sections).

This corresponds to the first two items on the Agile Manifesto: Individual and interactions, and Working software.

However, I find the current implementations of the realizations of the Agile Manifesto lacking. Without doubt, they are well mend. But their implementation in many projects I´ve visited is strangely at odds with reality.

Teams are suffering even more from interruptions, because they now are trying to follow an ideal of not being interrupted. Customers are much controlled in their expression of how they think a project should continue. And teams have a hard time to assess how their code scores on an evolvability scale.

So my overall conclusion is: We need a next iteration of software development processes. XP, Scrum, Kanban are not the end, but just a beginning.