The Architect´s Napkin

Software Architecture on the Back of a Napkin
posts - 69 , comments - 229 , trackbacks - 0

My Links

News

Article Categories

Archives

Post Categories

From Agile to Elastic

In my previous article I came to a couple of conclusions based on the reality of software development, or should I say “the nature of software development”? Here are the – to me - undeniable facts of what our industry is all about:

  • Customers hardly know, what they want. Any specification is inherently fuzzy and incomplete. What fits the customer´s needs can only be determined by actually trying it out. The customer can only recognize a running piece of software as acceptable.
  • Because customers hardly know, what they want, they are likely to change their mind at any time. Due to unforeseeable circumstances requirements get changed or re-prioritized. On top of that software needs to be corrected, since we can´t help but deliver it imperfect. There will always be bugs and other unintended deviations from what the customer wanted to express with a requirements definition.

From this follows, I think:

  1. The most important role in a software project is the Acceptor. The Acceptor should ask for working software as often as possible. The Acceptor is the driving, no, pulling force behind a software endeavor. The Acceptor´s interest is to keep the software as close as possible on target. Any deviation from a straight line to the goal of a fitting software is costing money. In Lean speak deviations are waste.
    Since the goal is not clearly known – see above – the frequency with which to check working software is crucial. The higher the better.
    (As a side note: It should be obvious, that to ask for an estimation on when requirements will be implement is absurd for several reasons. Firstly, such an estimation will always be just rough; we´re in the business of development, not reproduction. Secondly, it´s an estimation for something fuzzy and incomplete. Thirdly, until the date is reached an unknown amount of waste will have been produced and there will be no chance of changing one´s mind. But I digress…)
  2. A high frequency of checking for acceptability leads to a high rate of change requests. Yesterday’s plan will be today’s rubbish. In addition reports from support sooner or later will call for unplanned changes in the course of software development. The second most important property of a software development process thus is high responsiveness. It has to embrace ad hoc requests. Somehow the seemingly contradictory has to be accomplished: smooth flow with regard to strategic goals and constant change.

Scrum and XP come from the world of the waterfall. They were born from frustration with the waterfall. It´s only natural they still contain traces from their predecessor process. This trace is the long iteration. It might not be long compared to waterfall milestones measured in months or years. But it´s still long with regard to how much can go wrong during the several weeks of an iteration.

Kanban comes from machine production. Machines and their parts are copies; they are unchanging manifestations of an already existing design. The variability in a reproduction business – although it might be high – hardly is as high as the variability in software development. Especially when interruptions are taken into account.

XP, Scrum, and Kanban have their origin outside/before software development. Maybe we should try to come up with a process from within software development reality? How would that look like?


A new set of priorities

A truly software oriented process would need to embrace the idiosyncracies of software development. Let me phrase that in the manifesto manner we´ve come to love. It´s my “Elastic Manifesto” for today:

•    Acceptance over specification
•    Progress over completion
•    Reactivity over commitment

Customers are interested in just one thing: working software, which means: value generating software. So whatever helps to satisfy this interested, we should do.

Remember: We´re talking about software here. This stuff is supposed to be most flexible, limber. So whatever mankind has learned with regard to much, much harder, inflexible, rigid stuff has to be taken with a grain of salt. Therefore we should start at the other end. We should start with what inevitably makes software development special.


Acceptance over specification

Whatever is just specified, whatever is just implemented is vain – until the customer accepts it. Only when money flows due to acceptance is software development successful. Only then a software requirement can be considered done.

With all the talk about pull-based software development we´ve to acknowledge, that true pull only exists if the customer is pulling first. So where is this true pull in Scrum, where is it in Kanban? I´m unable to see it, sorry.

Kanban is agnostic about acceptance. It just states, how a process should be organized at all. Look at various Kanban boards on the web and you´ll hardly ever see “Acceptance” as the final column. You´ll find QA, sure. But QA is not the customer. QA does not release any money even if they deem a piece of software without a flaw.

And then: Where´s the pull in Kanban anyway? No kanban is passed upstream. Kanban is more about suction than about pull. If there is a free place in a queue, it sucks in a done element from the previous process step. But this item has not been produced on demand. There is no explicit demand in Kanban; it´s implicit.

Putting Acceptance over specification is about producing explicit and very tangible demand, though. A specification of 500 pages is no demand. It´s just a document sitting on a shelf. Demand is, when a manager rushes into the team room questioning the team why the feature he ordered yesterday to please his golfing buddy isn´t ready yet.

The bitter truth is: Any change to how a team develops software will be difficult unless it´s positively tied to money. A software development team is not serving quality, it´s serving money. And money is only released upon acceptance.

Peter Drucker said: “Culture eats strategy for breakfast.” So if there is no explicit and very tangible culture of acceptance it´s very hard to implement any change not tied to money. That specifically pertains to changes costing money but not obviously generating more of it.

Suggestions to reduce expenses are always welcome. Adding another developer to increase productivity or buying a tool to achieve the same is also welcome. But what about TDD or refactoring or time for learning? They require spending money, but it´s not obvious whether they will help generate/save any.

And even though I like Clean Code, Refactoring, TDD etc. I have to admit: they are just tools. They are only justified if they help to please the customer. But how to determine that without putting acceptance at the heart of the development process?

The literature focuses on how to write specifications and being available for clarifications during a sprint. Acceptance is just a “necessary evil” at the end of a sprint after several weeks of work on several features. Or it´s delegated to QA. But as I said: QA might do what´s called acceptance testing; but QA does not release any money. So we better call QA´s job “preliminary acceptance testing” or “acceptance preparation”.

So how can acceptance be put at the heart of the development process? How can an Acceptor be put before the software development cart? This is done by setting delivery dates first. A delivery date is even more important than the requirements definition. A delivery date should be present before any requirements are talked about. And it has to be a very tight deadline. The Acceptor needs to be a strict taskmaster. He has to exert a constant pull on the team. He has to constantly ask: “Where´s the next increment I can check?” Again: This is more important than having a specification. Why? Because in the end what´s really producing value for a customer is what gets delivered and accepted, not what got specified and delivered.

Also, only by acceptance are specifications vindicated as not being waste.

Most ailing software projects I´ve seen are lacking strict, constantly pulling Acceptor. Many stakeholders are eager to push requirements into the backlog. But this not matched by an eagerness to pull working software from the team to check, if these requirements are met. Ask any stakeholder telling you a user story when and how often she´s going to be available for acceptance testing. You´ll not be surprised to either hear, “Leave me alone until you´re done. You´re the expert, you know when the software is done.” or “This should be the job of QA. Why else are we talking about acceptance criteria?” In the end, though, the same person will be the first to complain about how the software delivered misses the mark. And this will be very likely weeks if not months from the user story conversation.

How can a team possibly improve under such circumstances? It´s very hard. Because whatever it does is not tied to feedback from the source of money. If stress rises, newly learned tricks would quickly be forgotten, because there is no link to the money.

Without an Acceptor constantly pulling at the team with his finger on the money-release-trigger, changes to a team´s behavior beg the question of relevance. But if the Acceptor can´t help but say, “Wow, you stand firm while I´m pulling at you. You constantly deliver value; you don´t budge when I give negative feedback; your reactions are prompt. Here´s the money you truly earned.” Then and only then changes are worthwhile.


Progress over completion

Currently software development is focused on completely implementing any specification. A user story (or use case) is detailed into features, and then features are fully implemented in one go if possible. Only if a feature is estimated to take longer than a couple of weeks it is further cut up to be delivered in parts. The ideal is, though, to finish a feature during a single iteration. Get it done, and then get on to the next feature.

This to me seems in contradiction to two undeniable facts: specifications are of notorious low quality and customers have constantly changing priorities.

The quality of a specification cannot be determined before Acceptance. The understanding of the specification by the team cannot be determined before Acceptance.

So why should a team strive for completion of something of unknown quality? Why should a team wait for feedback on its understanding of a specification until its fully implemented?

Please note the difference between talking to a customer for clarification during an iteration and Acceptance. A team does not earn money by asking questions; it gets only paid after Acceptance. Asking a customer questions is some form of “belated specification” or just a “specification refinement”; it does not change the basic nature of specifications as being a priori descriptions. Only working software, software trying to deliver value, something tangible a customer can give feedback about is the ultimate specification. That's what makes software development so special.

Neither a customer nor a team thus should seek completion. Because why should something of questionable value be completed at all? Much better than completion is Progress, constant, sustainable, valuable Progress.

The notion of Progress might lack a completion date. But instead it promises true value. Value added is part of the definition of Progress.

Also Progress allows the customer to change her mind at any time. Value exists only in the eye of the Acceptor. Whether it´s of higher value to refine a feature beyond the current specification or to switch to a different feature or to even stop work on a feature before it has been completed according to the current specification… that´s a decision a customer should be able to make at any time.

Think of all the money that could be saved if features were not completed but only implemented up to a “good enough” level. What good enough means cannot be determined a priori; only acceptance and usage will tell. But if features are always fully implemented before acceptance, there is no (official) opportunity to change direction or cut it off before.

In current agile development processes feature implementation is a ballistic flight. Features are fired off at iteration start and hopefully hit the target at the end of the iteration.

What software development should look like, though, is the flight of a Cruise Missile. Each and every feature should be constantly monitored for its progress. Checking every couple of weeks is not enough. Otherwise, if the priorities change, there is no process conformant way to steer the feature. Despite current iterative practice customer needs are deemed to be pretty immobile targets. This to me seems contrary to project reality and customer needs.

Feature development needs to be under constant scrutiny by Acceptance. This means switching from a focus on completion to a focus on “mere” Progress.

Progress means a constant dialog between development team and customer. The team is telling the customer how it understands her needs by delivering small increments of working software. The customer tells the team what she needs by giving feedback on small increments of working software. It´s all about information exchange according to Gregory Bateson, it´s about differences which make a difference.

Customer as well as development team will benefit from a switch from completion to Progress. Progress means less stress for the team, Progress means more flexibility for the customer, Progress means more ROI for the customer.

Completion then becomes a non-issue: a feature or a whole software project is complete once enough value has been created, when it´s good enough – or when the project has run out of time/money. But until then maximum value will have been generated.


Reactivity over commitment

From Acceptance pulling on the team follows a switch of focus on Progress. And from focusing on Progress follows the necessity to constantly react to changing priorities.

The customer´s current will for completion is matched by the teams commitment to actually complete. Therefore is does not make sense anymore to commit a team to completion once the customer favors Progress. Reactivity becomes more important than commitment.

Value can only progressively be produced if a team is able to smoothly react to all the different categories of requests fire at it. Already today customers value responsiveness higher than completion. Responsiveness immediately builds trust. Watching people interact with software freezing during an operation easily proves this: they quickly get impatient and doubt the software is still trying to produce value for them.

That´s why we love operating systems with preemptive multitasking. That´s why we use multi-threading to hide the latency of an operation.

However, current development processes favor teams to be left alone until they have fulfilled their commitment. A Scrum sprint must not be broken by switching priorities midway. The very notion of breaking a sprint is testimony to this attitude. And once a feature has been picked up in Kanban by a process step, this process step must not be interrupted for fear of increasing the work in progress (WIP).

A sudden change in the trajectory of development is not really part of the picture of Agility, even though one could think of such Reactivity to be core to the notion of agility.

Reactivity seems to be in stark contrast to what´s beneficial for good software. For years we´ve heard about the bad effects of human multi-tasking. For years we´ve been admonished to help developers focus.

But maybe that´s one of the reasons why Agility hasn´t been so easily adopted by all teams? Maybe its current variations are too far from reality, from what´s necessary to attract money? Customers don´t care about focused developers. They care about value. And value to them does not only mean completed features but also catering to their whims.

Of course teams need to strike a balance between Reactivity and focus. But my guess is, we can still improve on Reactivity while not going insane by constantly switching focus or letting code quality drop even further.

For a start just a simple change of mind is necessary: If someone comes in with a request, welcome him. Get the request triaged. Each and every request signifies how valuable the software is today for someone, and how it would be even more valuable tomorrow with the request implemented.

Only after you´re able to welcome every request and not feel flooded or irritated, find a way to increase the Reactivity of the team. It needs to get into a mode where is can naturally adapt to high request frequency variations and high request priority variations.


Summary

Yes, I guess we need to arrive at a software development process from the idiosyncrasies of software development. This means its form should follow the money, which comes from Acceptance, and the essential volatility of specifications as well as customer priorities.

We need a more elastic process to not break under the pressure of request frequency variations and priority changes.

Current agile processes are old school in so far as they just shortened former waterfall iterations. It´s now weeks instead of months or years. But how many weeks should an agile iteration last? Well, it depends – that´s the common answer. From one to four weeks to be more concrete.

This to me looks like growing a social hierarchy down from the root. Which is bad because higher levels in the hierarchy beg legitimation from their base.

Why not turn this around? Why not grow iterations from the base, from the smallest timebox of reasonable size? That´s what I´m going to talk about in my next article.

Print | posted on Friday, December 16, 2011 2:19 PM | Filed Under [ Thinking outside of the box Spinning ]

Feedback

Gravatar

# re: From Agile to Elastic

Not to be rude, but it looks like you don't really understand things like XP and don't actually understand where they came from or how they developed ( certainly not from waterfall, XP came from how Kent went about coding influenced by his smalltalk background where everything was iterative and adaptive based on feedback ).

XP, its not a spec you work on, and a "user story" is a 'promise for a conversation' where when you work on the story you work with the person who has the need and flesh out what they want.

Acceptance Testing is the final part of that process where, after working with the customer to build what they want. Everyone stands back and goes, 'is this good?'

What you are talking about is handled by XP teams very easily.
12/21/2011 11:00 PM | Keith Nicholas
Gravatar

# re: From Agile to Elastic

@Keith: As you can imagine, putting a phrase like "Not to be rude" in front of a judgement like yours does not really make it less rude. But anyway...

I perfectly aware of where Kent Beck "came from". But this background of his does not invalidate the context (!) in which software development was done back then. I don´t know how old you are, but I´m old enough to still feel what the mainstream back then was. And it´s that mainstream which motivated Kent Beck and others to try something different.

So when I write about XP´s connection to the waterfall I mean exactly this: XP was developed before the background of an unsatisfactory process, no, even because of this unsatisfactory process.

Also I perfectly well understand that XP includes acceptance tests. But as I said: acceptance tests run by some software developer is not Acceptance how I mean it. Unless at least running an acceptance tests successfully means money gets released by the customer.

But still: Who´s pulling at the team to release software frequently with all acceptance test green? Iterations in XP are long. There is no real pulling force to acceptance at the end.

And look where Kent Beck stands today. He certainly has evolved. But has the general XP practice?

Finally: Obviously, by many XP is not deemed enough of a software development. How else to explain the existence of Scrum and Kanban. So please leave room for further suggestions.
12/22/2011 8:15 AM | Ralf Westphal
Post A Comment
Title:
Name:
Email:
Comment:
Verification:
 

Powered by: