The previous article showed you how to easily parallelize flow operation execution with NPantaRhei (download the sources from github).

Once operations run concurrently, though, you´ve to pay your dues. Benefits or concurrent execution like higher performance, lower latency, or increased throughput come at a price. What that means conceptually e.g. for access to shared data, I leave to you to research. Here I want to show you how to practically handle errors in such flows. Because error handling cannot rely on the usual try-catch statement, you need to take care of it in your flow definition.

Causalities: Contexts for Exceptions

The question is: How to catch exceptions when there is no call stack anymore? You could do it by providing each operation with its own exception port and catch the exceptions yourself. This is ok, if you don´t want to check on too many operations. But what if an exeption occurs in an unchecked operation? One that you thought would not fail?

Wouldn´t it be better to just be able to say: “From here on, however the data flows, if an exception occurs, it should be handled such and such.”

That´s what try-catch does in a call stack world. And that´s what CCR causalities do in a concurrent world.

Causalities have been introduced by Microsofts Concurrency Coordination Runtime of ill fate. It was a great approach to more manageable concurrent programming, but did not gain much attention due to being buried inside Microsoft Robotics Studio. Nevertheless the concept stuck with me, and so I introduced it to the Flow Execution Engine.

Think of a causality as a very flexible sleeve or tube around a flow:

All operations inside this sleeve share a common exception handler. If an exception occurs it is caught by the causality and relayed to an exception handling operation. Nothing has to be changed within the operations inside the causality sleeve. The information on where to pass an exception is out of band. Any exception is caught by the Execution Engine which checks if a causality is active, and if so passes the exception wrapped inside a FlowRuntimeException to the causality´s exception port.

Once switched on, causalities stretch across the flow regardless where messages flow – until they are switched off. Even though the above drawing shows the causality as part of the flow it is in fact part of the messages flowing. When a message passes through the standard operation to switch on a causality, the causality is attached to the message. And when the message is worked on by an operation, the causality is copied to any messages produced by the operation. But that´s technical details. Better to think of a causality as a sleeve stretched over parts of a flow.

Like try-catch clauses causalities can be nested. Switching on a causality in fact is pushing a causality on a stack:

An exception occurring in operations C,E,F will be relayed to the red inner causality´s exception port, an exception in A,B,D,G will be relayed to the green outer causality´s port.

Please note: A causality is just a context for exceptions. It catches exceptions in any encompassed operations and relays them to a port. What you do with the exception is your business. The Execution Engine won´t halt because an exception occurred; it will continue processing messages. So if you want to change flow execution due to an exception, you have to build that into your flows.

Putting Causalities into a Flow

Back to the sample flow to show you, how to implement causalities using the Flow Execution Engine:

First define the streams flowing to/from the causality operations. To push the causality on the stack, just use the push causality operation´s name. The output port to connect to the exception handler is called “.exception”, though.

fr.AddStream(".in", "Pushc");
fr.AddStream("Pushc", "Find Files");
fr.AddStream("Pushc.exception", "Handle Exception");
fr.AddStream("Find Files", "Count Words");
fr.AddStream("Count Words", "Popc");
fr.AddStream("Popc", "Total");
fr.AddStream("Total", ".out");

Pushing and poping causalities is a matter of standard operations. Call them anyway you like. You can even reuse them in different parts of the flow:

var foc = new FlowOperationContainer()
    .AddFunc<string, IEnumerable<String>>("Find Files", Find_files)
    .AddFunc<IEnumerable<string>, IEnumerable<int>>("Count Words", Count_words)
    .AddFunc<IEnumerable<int>, Tuple<int, int>>("Total", Total)
    .AddPushCausality("Pushc")
    .AddPopCausality("Popc")
    .AddAction<FlowRuntimeException>("Handle Exception", Handle_exception);
fr.AddOperations(foc.Operations);

The exception handler is a method accepting a FlowRuntimeException:

void Handle_exception(FlowRuntimeException ex)
{
    …
}

Exceptions flow like any other data. Once the are caught by a causality you can process them in their own sub-flows.

Handling Unhandled Exceptions

If you don´t set up causalities or there are parts of your flows not covered by them, you can still catch exceptions. Just register an handler with the UnhandledException event of the Execution Engine:

fr.UnhandledException += ex => {…}

It´s that easy.

Just remember: Regardless of how you catch exceptions, they arrive on some background thread. So if you want to display some information on them in a GUI, you probably need to switch processing to the GUI thread. Use MakeSync() on a causality exception handler operation; do it your own way for unhandled exceptions.

My previous article introduced the Flow Execution Engine NPantaRhei (download here from github). It showed how to define a data flow and register its operations.

By default such flows are executed synchronously and sequentially – although in the background with regard to their initiator. But it´s easy to parallelize execution of operations. Here´s a first suggestion for how to run operations concurrently in the example given in the first article.

The dots in the operations signify asynchronous operation:

• A single dot means, the operation is running on its own single thread. Messages arriving do not hold up other operations. They are executed in the order they arrive.
• Two dots mean, the operation is running on multiple threads. Messages arriving do not hold up other operations. They are executed in parallel. Although execution is started in the order they arrive, the order of results might be different.
• No dot means, the operation is running just on the Execution Engine thread.

What the design above is supposed to mean is: There are two time consuming operations, finding all relevant files and counting the words in them. These operations should run concurrently so counting words can be started before all files have been found. And counting words in one file can be done in parallel to counting words in another file.

Totalling the word counts is not performance sensitive. It need not run on its own thread.

As easily as the above design can be understood and as simple as it could be implemented – it won´t work. Find Files and Count Words currently return a list of results. That´s not compatible with running concurrently. They need to be switched to stream output: Find Files needs to output a stream of filenames so each one can be processed on a different thread by Count Words. And thus there is no single Count Word method call running, but several, so Count Words can no longer output a list, but needs to switch to a stream too.

Totalling, however, needs to work on a list of word counts, or at least it needs to know how many word counts to exepect; its purpose is to output a single result.

For these reasons the design has to change:

The Scatter operation turns the list of file names produced by Find Files into a stream. And the Gather operation turns the stream of word counts back into a list. Neither the output of Find Files nor the input to Total need to change. Just Count Words has to be switched from list processing to stream processing.

Here´s how to set this design up with the Flow Execution Engine:

using(var fr = new FlowRuntime())
{
    fr.AddStream(".in", "Find_files");
    fr.AddStream("Find_files", "scatter");
    fr.AddStream("scatter.stream", "Count_words");
    fr.AddStream("scatter.count", "gather.count");
    fr.AddStream("Count_words", "gather.stream");
    fr.AddStream("gather", "Total");
    fr.AddStream("Total", ".out");

    var foc = new FlowOperationContainer()
        .AddFunc<string, IEnumerable<String>>("Find_files", Find_files).MakeAsync() /*2*/
        .AddFunc<string,int>("Count_words", Count_words).MakeParallel() /*3*/
        .AddFunc<IEnumerable<int>, Tuple<int,int>>("Total", Total);
    fr.AddOperations(foc.Operations);

    fr.AddOperation(new Scatter<string>("scatter")); /*1*/
    fr.AddOperation(new Gather<int>("gather"));

    fr.Start();

    fr.Process(new Message(".in", "…"));

    Tuple<int,int> result = null;
    fr.WaitForResult(5000, _ => result = (Tuple<int,int>)_.Data);
    …
}

Please note…

• /*1*/ …how the Scatter and Gather operations are standard operations which can be instantiated manually.
• /*2*/ …how any operation can be made async using the fluent interface of the operation container.
• /*3*/ …how any operation can be made parallel using the fluent interface of the operation container.

There is no need to “take back” the parallel processing for Total. But if you need to interface with a GUI framework you might be compelled to switch an operation back to the GUI thread. You can do that by applying .MakeSync() to the operation (optionally providing a SynchronizationContext).

Summary

Flow execution is done on 1+n threads. Without any intervention messages are taken from an internal queue by the Execution Engine running on its own thread. Each message is passed to an operation. Execution of an operation results in 0..n output messages which are put into the internal queue. This all happens on the Execution Engine thread. Once the operation finishes the call stack collapses back to the message loop of the Engine which picks the next message from the queue etc. At any time thus there is only one operation executing. That´s the default.

But if you like, you can have operations running on their own threads. Make them async (1 thread) or even parallel (n threads) so they don´t hold up message processing by the Engine. It´s as simple as switching concurrency on using the fluent interface shown above.

But beware: Concurrent operations come with their own problems. Data races and shared data access conflicts need to be taken into consideration. The Execution Engines just makes it easy to implement your designs; you need to get the concepts right yourself ;-)

Because that´s not always easy and bugs will kreep in, I´ll show you how to handle execeptions in my next article.

How can data processing flows be implemented in an easy manner?

What I call data flow processing – or flow design (FD) - I´ve defined here and described here for example. (Although this might look like Flow-Based Programming (FBP) it´s just related. FBP is all about concurrent programming, but FD starts much simpler with synchronous sequential programming.)

And how such flows can be translated into a modern OO language like C# is described here.

Such translation is pretty conventional, although it might look strange. Event-Based Components (EBC) are executed directly. They are written in a 3GL imperative way.

This has several drawbacks:

• Execution cannot be controlled. Once started it just runs unto its end.
• Although EBC code can be generated from some FD notation it´s imperative and not easy to reverse model. To get the model behind the code is hard at least for the untrained eye.
• It´s not so easy to switch from sync sequential processing to async or parallel processing.

Due to these and other drawbacks I decided to switch flow execution from direct execution to managed execution. That´s what the Flow Execution Engine is about. It interprets a flow definition thereby allowing for much more dynamic and fine grained control over the execution. Here´s how it works…

Example Flow

Let me use an example as the guiding rail for my tour through the Flow Execution Engine. The problem to solve: compile all .txt files in a directory tree and count their words. The FD model for this could look like this:

1. The first operation finds all the files in a given dir tree defined by a path.
2. The file names are passed to the second operation which processes each file by counting its words.
3. The word count for each file is passed to the totalling operation which outputs the total number of words in all files found plus the number of files processed.

The initial path is passed into the flow via the .in input port. The result can be retrieved from the output port .out.

All operation are feeded through implicit input ports; their output ports are also implicit. No names for their input/output ports are yet necessary, because there is only one of them on each operation.

Encoding the Model

With EBC such a model would be translated into C# code like this:

var ff = new Find_files();
var cw = new Count_words();
var tot = new Total();

ff.Result += cw.Process;
cw.Result += tot.Process;
tot.Result += result => {…};

Each operation would become a class with a method (Process()) as input port and an event (Result) as output port. Both, flow as well as operations would be described using the same 3GL.

The flow execution engine uses a different approach. Since it executes flows in a managed way it needs a flow definition it can understand. The simplest form of such a definition is a list of streams between operations, for example:

.in, Find Files
Find Files, Count Words
Count Words, Total
Total, .out

A stream is what connects operations, the arrow in the above diagram. It´s described by an output port as its source and an input port as its sink. Since input/output ports are implicit for such simple operations as above only their names are used in the stream table. Which port is meant is obvious from the position of the operation: input ports are referenced if an operation is noted in the right column, output ports are referenced in the left column.

To initialize the execution engine with a list of streams like this is easy:

using (var fr = new FlowRuntime())
{
    fr.AddStream(".in", "Find files");
    fr.AddStream("Find files", "Count words");
    fr.AddStream("Count words", "Total");
    fr.AddStream("Total", ".out");
    …

Implementing Operations

The flow execution engine makes implementing operations easier than EBC. No classes are required anymore. Operations are functions or methods, if you like. This is trivial as long as there is one input port and no or just one output port on an operation. The implementations for the example operations can look like this:

static IEnumerable<string> Find_Files(string path)
{
    …
    return …;
}

static IEnumerable<int> Count_Words(IEnumerable<string> filenames)
{
    …
    return …
}

static Tuple<int,int> Total(IEnumerable<int> wordCounts)
{
    return new Tuple<int, int>(wordCounts.Count(), wordCounts.Sum());
}

Registration of the operations is done through a special container which wraps each function in an IOperation:

var foc = new FlowOperationContainer()
    .AddFunc<string, IEnumerable<String>>("Find files", Find_files)
    .AddFunc<IEnumerable<string>, IEnumerable<int>>("Count words", Count_words)
    .AddFunc<IEnumerable<int>, Tuple<int, int>>("Total", Total);
fr.AddOperations(foc.Operations);

IOperation is the interface all operations need to implement:

public interface IOperation {
    string Name {get;}
    OperationAdapter Implementation {get;}
}

public delegate void OperationAdapter(IMessage input, 
                                      Action<IMessage> outputContinuation, 
                                      Action<FlowRuntimeException> unhandledException);

This way the flow execution engine is independent of how you specifically implement your operations. The flow operation container serves as the adapter between your implementation and the engine. For the most common way of implementing operations – functions, simple methods – the container readily provides registrations methods.

If you want more flexibility, though, you can provide IOperation implementations yourself. The abstract class AOperation makes this easy, e.g.

class Total : AOperation
{
    public Total(string name) : base(name) {}

    protected override void Process(IMessage input, 
                                    Action<IMessage> continueWith,
                                    Action<FlowRuntimeException> unhandledException)
    {
        var wordCounts = (IEnumerable<int>)input.Data;
        var result = new Tuple<int, int>(wordCounts.Count(), wordCounts.Sum());
        continueWith(new Message(base.Name, result));
    }
}

Such an operation then is registered directly with the flow execution engine:

fr.AddOperation(new Total(“Total”));

Executing a Flow

One the flow is defined and all operations registered, executing it is straightforward:

fr.Start();

fr.Process(new Message(".in", "c:/"));

Tuple<int, int> result = null;
fr.WaitForResult(5000, _ => result = (Tuple<int, int>)_.Data);

Console.WriteLine("{0} words in {1} files", result.Item2, result.Item1);

1. Start the flow execution engine by calling Start(). It runs on its own thread in the background. That means a flow is always executed in parallel to whatever else your programm is doing. However, this does not mean, the flow operations itself are run concurrently with regard to each other. They still are executed sequentially.
2. Send messages to the flow hosted by the engine using Process(). Messages are tuples consisting of some data and a source port.
3. Collect any results from the Result event by assigning an event handler or wait for them like above by calling WaitForResult().

How does this work?

The flow execution engine takes the message sent to it via Process() and puts it in a queue.

The main engine thread runs in a loop taking the next message from this queue. This happens on its own thread.

It looks at the message port (source) and looks it up in the list of streams. For the initial “.in” it finds “Find Files” as the sink.

The message then is sent to each sink port. That means a task is created which combines the message with the sink operation.

Then the operation is executed by calling it with the message as parameter.

Whatever the operation produces as a result it wraps up in an output message assigned to an output/source port of the operation.

These output messages are enqueued to be picked up by the execution engine main loop.

Messages flowing to sink ports starting with a “.” (like “.out”) are sent to the execution engine´s environment. They can be picked up from the Result event. Please note they arrive on a different thread than the caller´s. To make it easier to use the execution engine in synchronous programming use WaitForResult() to pick up messages on the same thread which called Process().

Summary

So much for a first introduction to the basics of the flow execution engine. You can download the sources of it from github:  https://github.com/ralfw/NPantaRhei.

But wait, there´s more! In future articles I´m going to show you how to switch from sync to async and parallel execution of flow operations. And how to handle execeptions occuring during execution.

Spinning as described in my previous article is all about flow. Its premise is: flow can emerge when work is partitioned in small, evenly sized chunks processed in a smooth manner.

There is a constant input of requests to the development team. A backlog is filled with strategically important requirements, support is reporting bugs, feedback requires changes, management wants to see ideas realized on short notice. Under these circumstances any plan becomes obsolete within a day or two. Or a lot of effort needs to be invested to stick to the plan.

Spinning is different. It does away with planning. So there is no plan anymore that can become obsolete. Instead each day the team decides what to accomplish, which means how to deliver a small value increment to the customer at the end of the day.

A value increment is…

• …at least whatever a customer can give feedback on. In some form or another it demonstrates the team´s understanding of the requirements. This is of course done best with…
• …working software the customer can Accept and actually use.

Value is generated whenever the customer´s trust is deepened. This happens when she feels to be understood. “Those guys really know what I mean! Yeah, they can relate to my problems. They home in on a solution. I can see it.” Working software is the best way to build trust. But sometimes less will do, too.

Software development is a communication process. That means messages have to go back and forth between customer and team. Spinning is setting a minimum frequency for this: every day the customer has to give feedback and tell the team what to do next (request), every day the team creates something valuable (response) to elicit feedback from the customer. Whatever is necessary to sustain this frequency should be done. Whatever stands in the way to obtain this frequency needs to be cut away.

Constantly moving closer to satisfaction is the purpose of Spinning. The basic means for that is to satisfy constantly – even if just a little bit. Satisfaction and trust are not built milestone by milestone or sprint by sprint, but day by day.

Whatever requests are fired at the team get processed day sized chunks:

The team is viewed as a “request processor” running in preemptive multitasking mode. There are several tasks to accomplish: a backlog needs to be burned down, bugs need fixing, operations need to be helped… A team not only needs to decrease latency, but also to hide latency.

With Spinning the ideal is to assign all developers to a single request per day (WIP=1). See the first three days in this picture:

On day 1 a request gets finished, on day 2 work starts at the next request. It continues on day 3.

But this ideal cannot be held up every single day. Sometimes developers need to attend to emergencies. If triage can schedule processing them for the next day, all´s well. Then developers just get allocated during daily work organization to more than one request.

But Spinning always reminds the team: the more developers are assigned to a single request, the better. More developers working on a request lead to faster implementation. Yes, producing a feature can be “industrialized” by distributing work. That´s at the core of Spinning; that´s why planning is so important before implementation. It´s all about speed to get feedback on working software.

On the other hand is illusionary to assume a team can focus on a single feature for several days. Support, operations, the boss´whims always cause disruption. Therefore the focus horizon is just one day. Most sudden requests can be postponed until the next day to be sliced, analyzed, organized, and implemented in an orderly way.

Sometimes however requests really require immediate attention. Such interruption of scheduled request increment implementation needs to be avoided. Hence it might be necessary to allocate one or two team members to be on standy-by for such work. If nothing happens they can work on a learning project or maybe a non time critical spike solution. But if an emergency calls for attendance, they are there – and the rest of the team can still quitely progress until the end of the day.

What´s the difference between Spinning and most teams? Most teams work like this, don´t they? They try to find a balance between progress and emergency handling. Yes, true. But they do so less systematically. And they constantly feel bad about it. They are working under the impression, software development should be different. They should be able to focus more. They should be able to keep the boss out of the team room for a couple of days. They should not be available for support.

Well, true, that would be much better – but unfortunately that´s not what the current culture of most projects allows. Spinning tries to take the burden of feeling bad off the backs of developers. It makes interruption the norm. Because every day the team is starting with a fresh view on what´s in the request queue.

Sure, teams doing Scrum and Kanban can organize themselves in a way to follow a process, and at the same time cater to emergencies. And they do. Every day. But this is outside of the process. The process is about focus. (Scrum more than Kanban.) To deal with different levels of urgency the process has to be tweaked.

Not so with Spinning. Spinning embraces imponderability. It´s defining a basic process mode

• Focus on delivering customer value at the end of every day
• Understand, plan, implement, check software in a systematic way in small increments
• Understand, plan, check software as a team to reap the collective intelligence of the team members, and disemminate information in a natural way
• Keep the Work-in-Progress at 1 – which can be different 1 each day

This helps to achieve two seemingly opposing goals:

• React to changing priorities: Spinning satisfies the needs of different stakeholders (customer, support, manager) by daily allocating developers to requests according to what seems to be a valuable increment.
• High speed development: Spinning satisfies the need for quick feedback by producing value every day with as many developers assigned to as few requests as possible.

Think of Spinning as a software production machine taking in raw material of very different size, shredding it into thin slices, and then transforming these slices into a steady flow of small increments of working software.

Spinning is true to the values of the Agile manifesto:

• Individuals and interactions are highly valued. That´s why the team is supposed to work on understanding, planning, and checking collaboratively. This of course includes the customer. Also on the next level of abstraction the very frequency with which to elicit feedback from the customer during Acceptance is a testimony to this. Delivering working software and receiving feedback are interactions.
• Working software is at the core of Spinning. Every day a working software increment has to be delivered. That´s also why Acceptance is so important to Spinning. When working on small increments on short notice, documentation loses value. Even a backlog loses value.
• Customer collaboration is the very reason for Spinning. Acceptance by the customer is the most important activity during software development. Only in order to satisfy the customers strict Acceptance rhythm specifications need to be given. Specification follows from Acceptance, not the other way around. Also the high frequency of Spinning make contracts with concrete scope definitions and delivery plans obsolete, since the customer is supposed to steer the project on a daily basis.
• Responding to change is the second reason for daily completion of requirement slices. No work in progress should be on a shelf at the end of the day. That ways the team is open to respond to any change the next day.

Through an orchestrated effort of individuals Spinning creates a constant flow of working software in close collaboration with the customer by responding to change at any time.

Agility needs to get onto the next level – that´s what I tried to explain in my previous articles. After a reality check – what´s missing from Agile practice? –, and some general musings about how a next level of Agility could look like, here now some very tangible suggestions.

Crank up the frequency

Current Agile practice is suffering from too little attention to Acceptance. To change this, very, very clear Acceptance dates need to be set. Acceptance can only get into a real pulling mode, if dates are fixed.

However, a date in weeks or months from now has no real pulling force on a team. This is obvious whenever you see a team change its working mode during an iteration because iteration end/delivery date is coming closer.

Acceptance always has to be around the corner. That means Acceptance has to be each and every day. That´s a frequency exerting real pull power over a team.

Yes, I mean it: A team should deliver a value increment every evening.

That´s how fast the development process should rotate at the core, at the implementation level. Delivering every day should be the default, the norm, the ideal. After sprinting comes Spinning.

Every evening (or next morning) the Acceptor should stand in the door of the team room an ask, “What have you got for me today?”

Again: I mean it. Requirements can and should be so finely sliced to be able to deliver value on a daily basis. This is possible, but it might require some training and good will. So please suspend your skepticism; don´t think, “This is impossible!” – but ask yourself “What could the benefits be?” Because there are many benefits to be reaped:

• Benefit #1: The team gets feedback on a daily basis. What got implemented yesterday is checked today by the Acceptor – and immediately returned upon dissatisfaction. All mental state about what the feedback is about is still present in the team.
• Benefit #2: If feedback is immediate and only a small increment has been added, not much waste can have been produced. Daily Acceptance transforms a project from ballistic flight to steered flight.
• Benefit #3: Where Acceptance is daily, the customer can change his mind daily. The team is never caught in limbo. At the end of the day it´s done. So it can take up whatever challenge the customer proposes for the next day. The customer is most flexible in how he wants the software to evolve. Delivering on a daily basis for Acceptance make it perfectly ok for the customer to change his mind on a daily basis.
• Benefit #4: The customer can stop work on features at any time. No features needs to be completed according to spec. This can safe quite some money, if a “good enough” state is reached before 100%. This can be checked for with daily Acceptance.
• Benefit #5: Not only the customer can change his mind daily, also the team can. It can allocate its resources very flexibly every day. That way, even most emergencies need not interrupt the team; they just get postponed a couple of hours until the next “planning point”, which is not father away than tomorrow.

Yesterday´s waterfall software production was like shooting a movie. The script needed to fixed for production planning and shooting.

Today´s agile software production is like shooting a TV series. An overall plot needs to be pretty much fixed before production can start, but the script for each episode can be written later. Each episode is like a sprint, a season is like a milestone release.

Tomorrow´s elastic software production then looks like improvisational theatre. There is just a rough idea of what to present on stage. All else is decided on a moments notice. The piece evolves according to the interaction between the actors and the feedback from the audience.

Only cranking up the Acceptance frequency will lead to this kind of fluidity of development. And it´s the emphasis on Acceptance, which makes the difference. A daily build has been suggested since long. But a daily build does not equal daily Acceptance. It´s the daily involvement of the customer that counts. Daily Acceptance is about money. Because the money needs to be satisfied, and only money can exert any power on the software production process.

That´s why other benefits will only be realized once Acceptance is daily:

• Benefit #6: The team will only change toward the better if Acceptance looms over it on da daily basis. Change will only happen, if it can demonstrate a connection to the money. Such connection is ever more easy to see, the closer the customer is. He´s most close if Acceptance is daily.
  Should a team do pair programming, reviews, TDD, should it refactor? Only if a positive impact on its performance can be shown the answer is yes. Performance, though, is under most tight de facto scrutiny when Acceptance is daily. Only changes to improve the capability to deliver at the end of the day true value are worthwhile to undertake.
• Benefit #7: The code will only change toward better inner quality (aka evolvability), if it’s under constant pressure. Daily Acceptance puts it to the test daily. Every day unforeseen changes can be chosen for delivery in the evening. A code base not highly evolvable will break quickly under this kind of stress. That´s a feature of Spinning, not a drawback. It´s better to see code break in a couple of weeks or months instead of years. Fail fast is true for evolvability, too. If the code base is still small, it´s easier to change design/implementation habits that lead to its breakdown.

The Elastic development process

Spinning very specifically is about software development. Its framework knows about the stuff that gets produced. This is different from Scrum or Kanban, which can be applied to all sorts of efforts. Both are product or even industry agnostic. Spinning is not.

Spinning is ambitious. Daily Acceptance is easy to accomplish feat. Therefore Spinning needs to help as much as possible to reach this goal. Here´s the basic Spinning process. It´s a production process transforming requests into working software.

Every day the process is progressing, every day it rolls value out the door. Progress is achieved by two activities: develop and accept. First working software is developed, and then it´s accepted; and so on…

Development itself is an activity to “run in circles”, to be done iteratively over the course of a progress cycle: at least once, but maybe more often, depending on the request situation.

From the perspective of Spinning development is not a black box. Spinning has an opinion about how software development should be done:

1. At the beginning of development the team needs to understand the problem. Too often developers jump onto implementation before they fully grasped what is supposed to be implemented in the first place. This results in a contra productive mixture of problem solving and coding. Also problem solving that way is often done alone; the intelligence of the whole team is not leveraged.
2. After understanding comes planning. Planning is needed to make software development a truly collaborative effort. Without planning software development deteriorates to a group activity instead of a team activity. During planning the team develops its idea of a solution to the just analyzed problem. This greatly helps collective project ownership.
3. It´s only after planning that implementation can start. During implementation developers individually work on code to contribute their share to the overall progress.
4. Finally the implementation needs to be checked before it can be handed off to Acceptance.

Each of these fundamental steps again consists of sub-steps. Some of them are established and self-explanatory, some need clarification.

What understanding means, seems to be obvious. But in fact it is not. Teams often brush over understanding a request in order to get to coding more quickly. This can be seen in many Coding Dojos, too. Even the simplest Code Katas are not really understood before someone fires up an IDE and starts hacking away on a test; this leads to more refactoring than needed.

Spinning tries to slow down teams in this regard. High overall velocity can only be achieved if a team has a solid grasp of the problem.

• Understanding a request requires first to analyze it: what´s the goal, what´s given, which resources are needed, what technologies could be helpful, have similar problems been solved already etc.? These and other questions need to be asked. By analyzing the request usually also a first approach to solving the problem is found.
• To solve the problem and move it downstream towards implementation – keep the delivery at the end of the day in the back of your head! –, the request is sliced. It´s decomposed in sub-requirements until the team identifies at least on slice it is confident to be able to implement until the end of the day; or it has identified enough uncertainty to warrant further learning through a spike solution.
• Triaging is needed to select the request(s) or request slice(s) to tackle until the end of the day. Each day the priorities can change for the customer or other stakeholders.

The order in which to analyze, triage, slice is not that important. Sometimes triage needs to be done first to zoom in on a request to actually analyze. Sometimes slicing comes right after analyzing to produce chunks that can be triaged. Sometimes the activities are repeated due to a deepened understanding of the problem.

In any case, only after requirements have been analyzed, sliced and triaged should planning start. Yes, planning. Spinning needs planning. Think of the proverb: “If you are in a hurry, go slower.” Planning is not a waste of development time, but actually a prerequisite for high velocity. Also as should be obvious from Spinning´s high frequency: planning is not “big plan upfront”. It´s “plan a little, implement a little”.

Planning is done in three different modes. And of course planning involves the whole team.

• During planning the software architecture is designed. This means the team defines a structure for the software in non-functional terms. Of course this pertains only to the request(s) currently under development.
• Architecture is only a framework for functionality. Hence planning continues with designing the functional structure for the current request slice(s). That´s what the model is about.
• Finally, once the solution has been designed (in a coarse grained manner) it needs to be decomposed and assigned to the team members for concurrent implementation. Spinning means, all (available) team members are working on the same feature. Spinning favors the feature team over the feature developer; the goal is Work-in-Progress=1.
  That way collective code ownership is created in a natural manner. Daily standups become obsolete, because understanding and planning are done collectively, and implementation is planned and executed in a way so each developer knows her part as well as the larger context. Information is disseminated constantly during Spinning.

Whereas the team as a whole understands and plans, implementation is an individual task. Whoever wants to pair program is welcome, though. Compared to the other process steps, though, implementation is a single developer effort. That way the developers can reduce the latency of coding a feature by working in parallel.

Implementation consists of coding and testing. They are the responsibilities of the developers. No code without automated tests. Implementation manifests the architecture and the model in code. Since design and implementation take place on a daily basis, it does not matter much, if “bubbles don´t crash”. Design´s “bubbles” get translated so quickly into code, it´s almost as if developers were design diagram compilers.

Coding is led by explicit design. But coding of course is also designing – but on a more fine grained level than architecture and model. So if a developer wants to use TDD during coding, that´s perfectly ok. However the importance of TDD is diminished in Spinning (compared to XP), since it´s not the sole tool for designing software. Spinning favors explicit design before coding to underline the importance of a big picture mental model, and to support a collective understanding of the design.

The final development step is checking the code quality. Implementation produces a software increment to provide value to the customer. That´s at least the intention of understanding, planning, and implementing a request slice. But does it succeed?

• A continuous build process first formally checks the quality of the result of the previous development steps. Does the code compile, do all tests run flawlessly, can it be deployed to different target platforms?
• As important a continuous build is, it´s not enough. A code review is needed for several reasons. Firstly, only a code review can check if the design has been faithfully implemented. If deviations are not spotted and discussed immediately, the quality of the software structure quickly deteriorated and no common mental model exists anymore. Secondly, a code review is highly effective in spotting errors and deviations from requirements. Thirdly, during code review the whole team is back together to deepen its collective project ownership and share knowledge. Fourthly, during a code review any conventions can be checked.
• Finally, after the build process and the developers have checked the code, quality can be assured by looking at the software from the outside. Some QA surely will be part of the automatic build process, but mostly there will be some work left for humans.
  To assure the team of the quality of its increment is an ongoing effort, although it looks like it´s just a process step at the end. Remember: understanding, planning, implementing, checking is done every day. And since people from QA belong to the team, they are involved from beginning to end.

In Spinning there is no “throwing code over the fence” like there is no “throwing a design over the fence”. A day in Spinning is mostly filled with the team members actually talking with each other by collectively understanding, planning, and checking the software.

 

Spinning only works if a group of developers really is a team – which is not the same. Or when there is no real team yet, Spinning helps to knit together the group members into a team. They almost can´t help but become a team, because otherwise the pull by daily Acceptance cannot be handled.

Summary

Spinning is serious about a couple of things:

1. Software development is a true team effort. A cross-functional team is needed.
2. Software development needs an explicit process to ensure high quality and constant flow. And this process needs to be specific to software development.
3. Software development is only as good as Acceptance is.

Feedback and flexibility are increased, and running development in a tight circle minimizes detrimental effects of explicit planning and designing while increasing code quality and speed.

How exactly you do your request analysis, model before coding, organize the implementation or do your reviews… that´s up to you. If you find UML, TDD, pair programming helpful, that´s fine. You´ll soon notice if these tools are useful to keep up with the high frequency of Spinning.

Also Spinning is agnostic to whatever you think you need to wrap around it. You can use Spinning within Scrum to iteratively home in on your sprint commitment. You can use Spinning within Kanban to decrease batch size. Call Spinning an Agile core practice, if you like. In the end, though… Spinning alone should do the job.

View the process steps as items on a checklist. Check them off each day. By doing so ask the team: “What have we done regarding X?” with X being analysis, triage, …, review, QA. Make this your daily reflection habit.

Sometimes you´ll find you do more understanding or modeling than is necessary for today´s goal: deliver working software at the end of the day. That´s fine. Keep the insights in the back of your collective head for tomorrow. Just be sure to progress every day. Sometimes your steps will be bigger, sometimes they´ll be smaller. It´s not the size of your progress steps that matters much, but to continually progress. Deliver value to the customer each and every day. Call some request slice done each and every day. That will make your customer happy; it builds trust; and you can go home relaxed each night. No pending issues, no unfinished work. Spinning thus not only tries to be very true to the Agile manifesto, but also to the Cult of Done manifesto.

In my previous article I came to a couple of conclusions based on the reality of software development, or should I say “the nature of software development”? Here are the – to me - undeniable facts of what our industry is all about:

• Customers hardly know, what they want. Any specification is inherently fuzzy and incomplete. What fits the customer´s needs can only be determined by actually trying it out. The customer can only recognize a running piece of software as acceptable.
• Because customers hardly know, what they want, they are likely to change their mind at any time. Due to unforeseeable circumstances requirements get changed or re-prioritized. On top of that software needs to be corrected, since we can´t help but deliver it imperfect. There will always be bugs and other unintended deviations from what the customer wanted to express with a requirements definition.

From this follows, I think:

1. The most important role in a software project is the Acceptor. The Acceptor should ask for working software as often as possible. The Acceptor is the driving, no, pulling force behind a software endeavor. The Acceptor´s interest is to keep the software as close as possible on target. Any deviation from a straight line to the goal of a fitting software is costing money. In Lean speak deviations are waste.
   Since the goal is not clearly known – see above – the frequency with which to check working software is crucial. The higher the better.
   (As a side note: It should be obvious, that to ask for an estimation on when requirements will be implement is absurd for several reasons. Firstly, such an estimation will always be just rough; we´re in the business of development, not reproduction. Secondly, it´s an estimation for something fuzzy and incomplete. Thirdly, until the date is reached an unknown amount of waste will have been produced and there will be no chance of changing one´s mind. But I digress…)
2. A high frequency of checking for acceptability leads to a high rate of change requests. Yesterday’s plan will be today’s rubbish. In addition reports from support sooner or later will call for unplanned changes in the course of software development. The second most important property of a software development process thus is high responsiveness. It has to embrace ad hoc requests. Somehow the seemingly contradictory has to be accomplished: smooth flow with regard to strategic goals and constant change.

Scrum and XP come from the world of the waterfall. They were born from frustration with the waterfall. It´s only natural they still contain traces from their predecessor process. This trace is the long iteration. It might not be long compared to waterfall milestones measured in months or years. But it´s still long with regard to how much can go wrong during the several weeks of an iteration.

Kanban comes from machine production. Machines and their parts are copies; they are unchanging manifestations of an already existing design. The variability in a reproduction business – although it might be high – hardly is as high as the variability in software development. Especially when interruptions are taken into account.

XP, Scrum, and Kanban have their origin outside/before software development. Maybe we should try to come up with a process from within software development reality? How would that look like?

A new set of priorities

A truly software oriented process would need to embrace the idiosyncracies of software development. Let me phrase that in the manifesto manner we´ve come to love. It´s my “Elastic Manifesto” for today:

•    Acceptance over specification
•    Progress over completion
•    Reactivity over commitment

Customers are interested in just one thing: working software, which means: value generating software. So whatever helps to satisfy this interested, we should do.

Remember: We´re talking about software here. This stuff is supposed to be most flexible, limber. So whatever mankind has learned with regard to much, much harder, inflexible, rigid stuff has to be taken with a grain of salt. Therefore we should start at the other end. We should start with what inevitably makes software development special.

Acceptance over specification

Whatever is just specified, whatever is just implemented is vain – until the customer accepts it. Only when money flows due to acceptance is software development successful. Only then a software requirement can be considered done.

With all the talk about pull-based software development we´ve to acknowledge, that true pull only exists if the customer is pulling first. So where is this true pull in Scrum, where is it in Kanban? I´m unable to see it, sorry.

Kanban is agnostic about acceptance. It just states, how a process should be organized at all. Look at various Kanban boards on the web and you´ll hardly ever see “Acceptance” as the final column. You´ll find QA, sure. But QA is not the customer. QA does not release any money even if they deem a piece of software without a flaw.

And then: Where´s the pull in Kanban anyway? No kanban is passed upstream. Kanban is more about suction than about pull. If there is a free place in a queue, it sucks in a done element from the previous process step. But this item has not been produced on demand. There is no explicit demand in Kanban; it´s implicit.

Putting Acceptance over specification is about producing explicit and very tangible demand, though. A specification of 500 pages is no demand. It´s just a document sitting on a shelf. Demand is, when a manager rushes into the team room questioning the team why the feature he ordered yesterday to please his golfing buddy isn´t ready yet.

The bitter truth is: Any change to how a team develops software will be difficult unless it´s positively tied to money. A software development team is not serving quality, it´s serving money. And money is only released upon acceptance.

Peter Drucker said: “Culture eats strategy for breakfast.” So if there is no explicit and very tangible culture of acceptance it´s very hard to implement any change not tied to money. That specifically pertains to changes costing money but not obviously generating more of it.

Suggestions to reduce expenses are always welcome. Adding another developer to increase productivity or buying a tool to achieve the same is also welcome. But what about TDD or refactoring or time for learning? They require spending money, but it´s not obvious whether they will help generate/save any.

And even though I like Clean Code, Refactoring, TDD etc. I have to admit: they are just tools. They are only justified if they help to please the customer. But how to determine that without putting acceptance at the heart of the development process?

The literature focuses on how to write specifications and being available for clarifications during a sprint. Acceptance is just a “necessary evil” at the end of a sprint after several weeks of work on several features. Or it´s delegated to QA. But as I said: QA might do what´s called acceptance testing; but QA does not release any money. So we better call QA´s job “preliminary acceptance testing” or “acceptance preparation”.

So how can acceptance be put at the heart of the development process? How can an Acceptor be put before the software development cart? This is done by setting delivery dates first. A delivery date is even more important than the requirements definition. A delivery date should be present before any requirements are talked about. And it has to be a very tight deadline. The Acceptor needs to be a strict taskmaster. He has to exert a constant pull on the team. He has to constantly ask: “Where´s the next increment I can check?” Again: This is more important than having a specification. Why? Because in the end what´s really producing value for a customer is what gets delivered and accepted, not what got specified and delivered.

Also, only by acceptance are specifications vindicated as not being waste.

Most ailing software projects I´ve seen are lacking strict, constantly pulling Acceptor. Many stakeholders are eager to push requirements into the backlog. But this not matched by an eagerness to pull working software from the team to check, if these requirements are met. Ask any stakeholder telling you a user story when and how often she´s going to be available for acceptance testing. You´ll not be surprised to either hear, “Leave me alone until you´re done. You´re the expert, you know when the software is done.” or “This should be the job of QA. Why else are we talking about acceptance criteria?” In the end, though, the same person will be the first to complain about how the software delivered misses the mark. And this will be very likely weeks if not months from the user story conversation.

How can a team possibly improve under such circumstances? It´s very hard. Because whatever it does is not tied to feedback from the source of money. If stress rises, newly learned tricks would quickly be forgotten, because there is no link to the money.

Without an Acceptor constantly pulling at the team with his finger on the money-release-trigger, changes to a team´s behavior beg the question of relevance. But if the Acceptor can´t help but say, “Wow, you stand firm while I´m pulling at you. You constantly deliver value; you don´t budge when I give negative feedback; your reactions are prompt. Here´s the money you truly earned.” Then and only then changes are worthwhile.

Progress over completion

Currently software development is focused on completely implementing any specification. A user story (or use case) is detailed into features, and then features are fully implemented in one go if possible. Only if a feature is estimated to take longer than a couple of weeks it is further cut up to be delivered in parts. The ideal is, though, to finish a feature during a single iteration. Get it done, and then get on to the next feature.

This to me seems in contradiction to two undeniable facts: specifications are of notorious low quality and customers have constantly changing priorities.

The quality of a specification cannot be determined before Acceptance. The understanding of the specification by the team cannot be determined before Acceptance.

So why should a team strive for completion of something of unknown quality? Why should a team wait for feedback on its understanding of a specification until its fully implemented?

Please note the difference between talking to a customer for clarification during an iteration and Acceptance. A team does not earn money by asking questions; it gets only paid after Acceptance. Asking a customer questions is some form of “belated specification” or just a “specification refinement”; it does not change the basic nature of specifications as being a priori descriptions. Only working software, software trying to deliver value, something tangible a customer can give feedback about is the ultimate specification. That's what makes software development so special.

Neither a customer nor a team thus should seek completion. Because why should something of questionable value be completed at all? Much better than completion is Progress, constant, sustainable, valuable Progress.

The notion of Progress might lack a completion date. But instead it promises true value. Value added is part of the definition of Progress.

Also Progress allows the customer to change her mind at any time. Value exists only in the eye of the Acceptor. Whether it´s of higher value to refine a feature beyond the current specification or to switch to a different feature or to even stop work on a feature before it has been completed according to the current specification… that´s a decision a customer should be able to make at any time.

Think of all the money that could be saved if features were not completed but only implemented up to a “good enough” level. What good enough means cannot be determined a priori; only acceptance and usage will tell. But if features are always fully implemented before acceptance, there is no (official) opportunity to change direction or cut it off before.

In current agile development processes feature implementation is a ballistic flight. Features are fired off at iteration start and hopefully hit the target at the end of the iteration.

What software development should look like, though, is the flight of a Cruise Missile. Each and every feature should be constantly monitored for its progress. Checking every couple of weeks is not enough. Otherwise, if the priorities change, there is no process conformant way to steer the feature. Despite current iterative practice customer needs are deemed to be pretty immobile targets. This to me seems contrary to project reality and customer needs.

Feature development needs to be under constant scrutiny by Acceptance. This means switching from a focus on completion to a focus on “mere” Progress.

Progress means a constant dialog between development team and customer. The team is telling the customer how it understands her needs by delivering small increments of working software. The customer tells the team what she needs by giving feedback on small increments of working software. It´s all about information exchange according to Gregory Bateson, it´s about differences which make a difference.

Customer as well as development team will benefit from a switch from completion to Progress. Progress means less stress for the team, Progress means more flexibility for the customer, Progress means more ROI for the customer.

Completion then becomes a non-issue: a feature or a whole software project is complete once enough value has been created, when it´s good enough – or when the project has run out of time/money. But until then maximum value will have been generated.

Reactivity over commitment

From Acceptance pulling on the team follows a switch of focus on Progress. And from focusing on Progress follows the necessity to constantly react to changing priorities.

The customer´s current will for completion is matched by the teams commitment to actually complete. Therefore is does not make sense anymore to commit a team to completion once the customer favors Progress. Reactivity becomes more important than commitment.

Value can only progressively be produced if a team is able to smoothly react to all the different categories of requests fire at it. Already today customers value responsiveness higher than completion. Responsiveness immediately builds trust. Watching people interact with software freezing during an operation easily proves this: they quickly get impatient and doubt the software is still trying to produce value for them.

That´s why we love operating systems with preemptive multitasking. That´s why we use multi-threading to hide the latency of an operation.

However, current development processes favor teams to be left alone until they have fulfilled their commitment. A Scrum sprint must not be broken by switching priorities midway. The very notion of breaking a sprint is testimony to this attitude. And once a feature has been picked up in Kanban by a process step, this process step must not be interrupted for fear of increasing the work in progress (WIP).

A sudden change in the trajectory of development is not really part of the picture of Agility, even though one could think of such Reactivity to be core to the notion of agility.

Reactivity seems to be in stark contrast to what´s beneficial for good software. For years we´ve heard about the bad effects of human multi-tasking. For years we´ve been admonished to help developers focus.

But maybe that´s one of the reasons why Agility hasn´t been so easily adopted by all teams? Maybe its current variations are too far from reality, from what´s necessary to attract money? Customers don´t care about focused developers. They care about value. And value to them does not only mean completed features but also catering to their whims.

Of course teams need to strike a balance between Reactivity and focus. But my guess is, we can still improve on Reactivity while not going insane by constantly switching focus or letting code quality drop even further.

For a start just a simple change of mind is necessary: If someone comes in with a request, welcome him. Get the request triaged. Each and every request signifies how valuable the software is today for someone, and how it would be even more valuable tomorrow with the request implemented.

Only after you´re able to welcome every request and not feel flooded or irritated, find a way to increase the Reactivity of the team. It needs to get into a mode where is can naturally adapt to high request frequency variations and high request priority variations.

Summary

Yes, I guess we need to arrive at a software development process from the idiosyncrasies of software development. This means its form should follow the money, which comes from Acceptance, and the essential volatility of specifications as well as customer priorities.

We need a more elastic process to not break under the pressure of request frequency variations and priority changes.

Current agile processes are old school in so far as they just shortened former waterfall iterations. It´s now weeks instead of months or years. But how many weeks should an agile iteration last? Well, it depends – that´s the common answer. From one to four weeks to be more concrete.

This to me looks like growing a social hierarchy down from the root. Which is bad because higher levels in the hierarchy beg legitimation from their base.

Why not turn this around? Why not grow iterations from the base, from the smallest timebox of reasonable size? That´s what I´m going to talk about in my next article.

Let´s get real about software development: It´s never going to be a quietly flowing river. Never. And that´s why the current approaches to software development like XP, Scrum, and Kanban will always cause pain.

Their basic assumption is you should be able to isolate a team for a while to work on features. Leave it alone during an iteration or a sprint to complete a set of features, or at least sit still until the current feature is done.

Certainly that´s what we all want as developers: being able to concentrate on creating value by implementing new features or at least improving existing ones. Until we´re done.

Project reality seems to differ, though. And it does not just differ because Scrum or Kanban haven´t been implemented faithfully. Project reality will always differ because customers and users don´t care about our longing for being left alone.

Get real about emergencies

Software development is not like some factory production where orders are worked on in a smoothly flowing manner until fulfilled. Software development is more like an emergency unit in a hospital. It´s constantly under fire from all sides.

Requests are coming in at all times. Resources are always limited. That means a software team has to use triage (http://en.wikipedia.org/wiki/Triage) to determine what to do. Three request categories seem appropriate:

1.    Requests that must be answered immediately.
2.    Requests that should be answered soon. If not right away, then tomorrow is still ok, but not much later.
3.    Requests that can be answered at no specific date. Tomorrow would be ok, also next week, maybe even next month.

Category 1 is about rare bugs causing heavy damage with customers right now or in the foreseeable future; show stoppers so to speak. They need to be taken care of within a minutes or hours. Whether certain requests justifiably are categorized as show stoppers or not, is of no importance. Someone high enough up the ladder thinks a request is in category 1, so the team needs to react immediately.

Naturally category 1 requests should be rare. More than once a week seems problematic to me. Not because such interruptions are to be avoided, but because such urgent requests either hint at very poor software quality or an undue relationship with the customer base.

Category 2 is about most bugs. They need to be cared for pretty quickly, but not immediately. Bugs are a threat to any customer relationship. Customers rightly assume bug free software – with bug meaning an unintended deviation from what a software developer was trying to deliver. A bug is the fault of the development team so it needs to take care of it as quickly as possible.

Category 3 is about new features, feature change requests, and corrections due to uncovered misunderstandings or deeper insight. No life is at stake if such a request is not processed immediately. It can be put on a backlog for prioritization.

Any project not just started is under fire with all requests from all three categories. In an early greenfield project, requests might just fall into category 3. But as soon as customers/users have put their hands on some (early) release, category 2 and 1 requests will start to come in.

That´s a fact no software development process should deny or neglect. Right to the contrary any software development process should revolve around this basic fact of software development reality.

My conclusion #1: We need a software development process, which institutionalizes request triage.

Get real about uncertainty

At the heart of Agile and Lean is the realization that requirements are notoriously poor. As much as we invest in requirements elicitation… the result still is fuzzy. I´d even say: Customers cannot specify what they want, they can only recognize if that, what software development releases to them, is what fits their needs.

Customers are poor at a priori definition, but extremely good at a posteriori judgment.

As software developers we thus live in a constant state of uncertainty. We just don´t know if what we´ve developed really is what gets accepted by the customer. Did the customer define his requirements correctly? Are they complete? Are the comprehensive? Does the customer know at all what´s necessary to solve her problem? Have we understood the requirements? Have we come up with a solution matching our understanding? Will the customer accept our view?

Because of this uncertainty Agility/Lean advocates iterative development with delivery of completed features. Whatever get´s released to the customer at the end of an iteration has value to him; he can give feedback, because some requirement has been implemented.

How long until such a release? Two to four weeks seem the norm. Sometimes it´s just one week, sometimes it just takes as long as it takes for a feature. Iterations usually are measured in weeks, not months, not days or hours.

This seems ok – until iteration timeboxes collide with project reality. Any request of category 1 or 2 is likely to cause a delay. Such requests naturally are a nuisance for any process requiring quite time to work on a plan laid out for a couple of weeks.

My conclusion #2: We need a development process, which embraces the undeniable reality of category 1 and 2 requests.

But not only are timeboxes of several weeks in contradiction with daily project life, they also still do not really honor the uncertainty of requirements. Think of it: A day 1 of an iteration you start working on a feature. You finish it by day 4. But the only gives feedback at the end of the iteration, maybe on day 14 or 21. That´s a long time since you checked off the feature in your head as done and got rid of your mental feature state. The feedback loop is quite long even in today´s agile processes. On average it takes at least half an iteration until you hear back from the customer.

Also, during an iteration several features are implemented. This means, whatever change request the customer might have at the end of the iteration, it can possibly affect all features under development within the same iteration. The longer an iteration the more likely waste is created.

My conclusion #3: We need a development process, which allows for almost arbitrarily quick feedback from customers.

And then think of it: If requirements are so uncertain, why should a team deliver whole features at all at the end of a multi-week iteration? Neither does the team know, if it interpreted the requirements correctly, nor is it clear to the customer, if she really needs all that which she found important many weeks or months ago during requirements elicitation.

If a process tries to deliver as much of a feature as possible in a multi-week timebox, there is hardly any chance for the customer to say, “Stop! That´s good enough.” Current processes implement requirements in a very coarse grained manner. Their ideal is completion, arriving at Done for a whole use case or feature. That seems like a relic from the old waterfall days, to me. It´s not very true to the fundamental realization of Agility: Requirements are notoriously uncertain; we need to deliver often.

My conclusion #4: We need a development process, which allows customers to stop development of features at almost any percentage of completion.

Get real about evolvability

Before Agility the biggest challenge for software development was technology. We struggled to fulfill the explicit functional and non-functional requirements of customers.

The next big challenge then was correctness. Also a non-functional requirement, but not often explicitly stated. Customers rightly assume software to be correct, i.e. bug free. There is no excuse for a crash or an incorrect calculation.

Since then, though, the world has moved on. There are tons of technologies and tools at our disposal. There are several systematic approaches to increase correctness of code from reviews, to automatic unit tests, integration tests, and acceptance tests.

So I daresay today´s biggest challenge for software development is different. It´s harder to solve, because to assess the quality of our solutions with regard to it, is more difficult.

What I see as the biggest current challenge is maintainability or evolvability. We need to do a much better job keeping our code flexible. Cost is rising too steeply over the lifetime of a code base. Implementing a feature in month 3 of a project might cost X dollars, implementing it in month 18, though, will cost much, much more.

There are several approaches to attack this problem like Clean Code, Agile Architecture, or just tried and true principles and practices from 60 years of software development.

Strangely, however, current software development processes do not address the challenge. Neither Scrum nor Kanban are interested in code evolvability. They might measure a decrease in productivity, but they have nothing to say about how to improve the situation.

XP might be different in that regard. But then… how effective can it be to have list of principles and practices for high evolvability, when evolvability is not put to the test?

Yes, I think that´s a common blind spot of all current development processes: they do not really put evolvability to the test.

And why should they? Isn´t evolvability a matter of code design instead of process? Indeed it is – but alas there is hardly a measure for the quality of code base evolvability. Forget about LOC per method or class, forget about Cyclomatic Complexity, forget about afferent coupling etc. Those are just numbers that must be interpreted. And this interpretation is highly subjective and relative. Let´s get real: We can´t look at any set of measurements today and positively say, “This code has high evolvability.”

Checking the functional quality of code, checking it´s correctness, checking its non-functional qualities… that´s all different and comparatively easy. Checking evolvability, though, is hard. In the end the proof is only in the pudding. If a code base is hard to evolve, then, well, its evolvability must be low.

To check the fulfillment of functional requirements, execute the program. To check non-functional requirements, execute the program. To check the evolvability, evolve the code.

Sure, design/code developed using XP, Scrum, Kanban is put through evolutions more often than in a waterfall process. But my perception is, that´s not enough. There are so many teams trying to improve evolvability by applying Clean Code principles etc. – but the results are still suboptimal. From that I draw my last conclusion:

My conclusion #5: We need a development process, which addresses today´s major challenge evolvability head on by putting much more pressure on a code base through shortened evolution cycles.

Summary

Current development processes are a big step forward compared to whatever was the norm before. They have done a lot of good to the state-of-the-art of software development. To me the major contributions are:

• We now know that requirements are very uncertain. The problem to solve is always poorly understood by customers and developers alike. To arrive at a solution we need to interact frequently.
• In order to home in on a solution, a development team needs to deliver artifacts to customers on which they can give feedback. Software thus is delivered in slices (vertical sections), not layers (horizontal sections).

This corresponds to the first two items on the Agile Manifesto: Individual and interactions, and Working software.

However, I find the current implementations of the realizations of the Agile Manifesto lacking. Without doubt, they are well mend. But their implementation in many projects I´ve visited is strangely at odds with reality.

Teams are suffering even more from interruptions, because they now are trying to follow an ideal of not being interrupted. Customers are much controlled in their expression of how they think a project should continue. And teams have a hard time to assess how their code scores on an evolvability scale.

So my overall conclusion is: We need a next iteration of software development processes. XP, Scrum, Kanban are not the end, but just a beginning.

Ron Jeffries challenged me to show how Flow-Design and Event-Based Components can help software development. This is the problem he posed in the Software Craftsmanship discussion group:

Solve bowling scoring. Here is the specification. Note that this is a simpler
version than the one Bob Martin often uses. I'll take questions if you have any.

  Given a list of the rolls of a legal game of ten pin bowling,
  which you may assume are provided without error or omission,
  produce the total, final, score of the game.

  Examples:
    Twenty zeros produce zero.
    Twenty fours produce 80.
    Twelve tens produce 300.

Since I don´t know much about bowling I consulted the KataBowling description as a second source. I haven´t done the kata before and have not looked at any of the kata solutions on display on the internet. That´s fortunate for a fresh and unbiased start – but it´s also a pain, since I simply don´t like bowling. But, well, I don´t want to complain. Here we go…

Understanding the problem

I believe in thinking before coding. And the first thing to think about is the problem. Before I can start to code I need to understand the problem which also means I have at least an idea of how to solve it.

It´s too bad I can´t show you the process of actually gaining an understanding of the problem. But what I can show you is data. Here´s a sample bowling game score sheet:

This shows you how I interpret the rules I read. Not just as a test case, but with some explanations as to why the scores are calculated like that.

The input to the solution I´m supposed to write would look like follows for this game:

It´s just the number of pins knocked down with each roll.

The combination of this list and the total score (here: 131) describes an acceptance test. Ron also provided me with some acceptance tests (see above).

Please note: Acceptance tests given by the customer are not really enough. You yourself should reply back to your customer with a couple of acceptance test suggestions. That´s what my game score sheet is about. Because only by defining your own acceptance test cases and showing them to the customer you can be sure to demonstrate you understand the problem.

If you explain something to someone – e.g. basic mathematical operations like +, –, *, / – and ask her, “Do you understand?” and she nods, that means nothing. You can only know if somebody understands what you´re saying by getting him to explain to you, what she understood or ask her questions.

The above description/acceptance test case is my reply to Ron. Hope he agrees I understood the problem correctly.

For the moment let me assume I did. What´s next?

“User Interface”

After I made sure I understand a problem, I like to clarify how the customer (here: Ron) wants to interact with my solution. Software is about transforming input into output. So how is the input provided? How should the output be returned? Often this entails user interaction and you need to talk with your customer about some kind of user interface. In this case, though, no UI is needed; at least Ron has not mentioned any. So I assume he´ll be satisfied with some kind of API. (As is the case for most Coding Katas.)

The API I´d like to suggest is this (Warning: I´ll be using C# as my solution language):

public class BowlingGame
{
    public static int CalculateTotal(IEnumerable<int> rolls) {…}
}

Using it would look like this:

var total = BowlingGame.CalculateTotal(new[]{3, 4, 5, …, 10, 3, 7, 5});

Acceptance Test Code

With acceptance tests and a UI in hand, I set up acceptance test code. This makes sure I do not deliver a solution not meeting the minimum criteria agreed upon my the customer. Employing NUnit this can look like this:

[Test]
[TestCase("00000000000000000000", 0)]
[TestCase("44444444444444444444", 80)]
[TestCase("AAAAAAAAAAAA", 300)]
[TestCase("192837465555647382915", 154)] // all spares
[TestCase("000000000000000000195", 15)]  // focus: just one more roll after spare in 10th frame
[TestCase("000000000000000000a34", 17)]  // focus: two more rolls after strike in 10th frame
[TestCase("3451a45267354aa375", 131)]    // acceptance test case from blog entry
public void AcceptanceTest(string rolls, int expectedTotal)
{   
    var total = BowlingGame.CalculateTotal(String2Rolls(rolls));
    Assert.AreEqual(expectedTotal, total);
}

The first three test cases are Ron´s. Then follow additional ones I derived from my understanding of the rules.

As soon as my solution delivers correct results for the acceptance test input data I´m ready to ship.

If I´d continue with TDD to design the implementation, I´d start from here. But I don´t. I continue with some more thinking…

Modelling the solution I

Instead of sitting down and code I like to close in on a solution using a stepwise refinement process. Here´s my first step; I draw the solution on the highes level of abstraction possible. The whole solution is a single black box:

There is a list of integers flowing into the one method representing the solution; and a single integer is flowing out of it as the result. Seen from far away that´s how the solution works, how in fact any solution works: rolls are transformed into a total score, input is transformed into output by a single action.

Given an action I can decide if I want to switch to coding mode. If I feel the solution will be short and I feel comfortable writing it down, I´d switch. But if not I´d rather continue modelling with a graphic language. It´s so much easier to change a mental model of a solution if it´s just a diagram.

In this case I decide to continue modelling not only because this is the purpose of the exercise, but also because I already have a clear vision of a more detailed design. Understanding the problem lead me to distinguish three operations:

Calculating the total score consists of adding the pins in each frame, adding bonus points for spares and adding bonus points for strikes.

I don´t think these operations are too far fetched. I find them pretty obvious from reading the requirements. I´d consider me not understanding the problem if I had no idea of these aspects of a solution.

But since I´m very sure these operations contribute towards the solution, I use them to refine the all encompassing action:

Now there are three functional units to put “domain logic” in – and one at the top just wiring them together. This is a stratified design: an operation on a higher level of abstraction is assembled from operations on a lower level of abstraction. A very well known approach to build complicated stuff all over the world.

But wait: there is a list of integers entering the root functional unit, but a list of frames go into the first operation. This does not look right. Where are the frames coming from? And where is the total score calculated? Two more functional units are needed:

This looks better: consistent and complete. This is even not very technical; I could explain it to the customer to make clear what kind of solution I envision.

Should I start coding now? I guess that would be ok. All functional units seem to be quite small. And they represent crucial domain terminology.

The basic idea of this solution is to transform the initial list of rolls into a set of frames – then enrich these frames with a score – and finally sum all frame scores. Hm… this sounds good. I should go back to the model and make it mirror this straightforward explanation:

That´s jojo-modelling, I´d say :-) First top-down, then bottom-up.

Now I can explain the solution to my fellow developers on three different levels. It´s complete on each level, but each level is lacking detail. That´s on purpose. Modelling with Flow-Design is about abstraction; and abstraction is about hiding details. That´s why Flow-Design does not want to duplicate what programming languages do. It´s not flow-charts, it´s data-flow. It´s not imperative, it´s declarative. The transformation depicted will magically happen like transforming data from sectors on a hard disk into records in memory magically happens when using SQL.

And what about the data? Usual object orientation start with focusing on the data.

With Flow-Design data is not the focus even though it´s about data-flow. Sure, in the end the details of the data flowing needs to be specified. But usually data is not the problem; how data should be structured mostly is pretty obvious. That´s probably one reason why you mostly start by modelling data structures: you feel comfortable, you get something done.

In the end, though, data is not really the problem. Transformation is. As programmers we´re hired to implement transformations. And since a well known advice is to start work with higher risk tasks I´d argue it´s good advice to start with the transformations when programming. Either data structures are well known – or transformations drive data structures (like TDD is supposed to drive design).

To start programming by identifying data classes from nouns in a requirements document might even be premature optimization. So be cautious about it.

But, yes, I need to define Frame before I start coding. Here it is:

class Frame
{
    public int[] Rolls = new int[2];
    public int Score;
}

More´s not necessary, I´d say. It´s devoid of functionality. Why? To be honest: I don´t know which functionality I should attribute to Frame. What´s the responsibility of Frame except to hold data? Responsibilities I feel sure about are modelled in the above diagram. To me it feels very natural to not force them onto frame.

Modelling the solution II

Although the above model seems to be up to the task, I don´t want to keep it as a secret: there is another way to model the solution. Maybe you even thought of this alternative first. It´s replacing the enrichment action sequence with parallel actions:

This is not to suggest there will be running anything in parallel (although it could). It´s only to make clear the independence of the actions to sum pins and bonuses.

Also note how frames enter the summation actions – but integers are leaving them. The list of frames will not get enriched. It´s just input to be traversed to calculate the output.

To me that sounds even better than the first model.

Nevertheless I´ll implement the first one because it lends itself nicely to a translation into very plain C# code as you´ll see. The second model I´ll leave to my colleague Stefan Lieser who´s working with me on Flow-Design and Event-based Components. I´ll describe it later in this blog.

For now I hope you feel with me at least a little bit how easy it is to reason about different approaches when looking at a picture. Imagine juggling models only in your head? Or imagine sitting at your IDE and coding away using TDD. You´ll most certainly focus on just a single solution. TDD will drive you into one direction without showing you the alternatives. Or if alternatives show up you´d need to experiment with them in code. Sure, that would be executable experiments – but it would also be quite tedious to explore them. You simply cannot type as fast as you can draw or think. (At least I cannot.)

So although “bubbles don´t crash” and there is no guarantee that the solutions are comprehensive I prefer to model them explicitly like this first. It´s sufficiently coarse grained to be swift. And it´s sufficiently fine grained as to make coding easier.

Translating the model

Enough scribbled. On to some code.

How should I start coding the first model?

Well, anyway I like. I can start top-down or bottom-up. The model tells me which functional units to code. I don´t need to find them out through refactoring, I know which ones are needed – at least at a certain level of abstraction.

So I randomly pick Add_pins_in_frame to implement first. Here´s my test – yes, I´m coding test-first:

[Test]
public void Calc_basic_scores()
{
    var frames = new[] { new Frame { Rolls = new[] { 1, 2 } },
                            new Frame { Rolls = new[] { 3, 5 } } };
    frames = frames.Add_pins_in_frame().ToArray();
    Assert.AreEqual(new[] {3, 8 },
                    frames.Select(f => f.Score).ToArray());
}

I chose to implement the action as a C# extension method. As you´ll see this will make the code very readable:

internal static class BowlingGameExtensions_Scoring
{
    public static IEnumerable<Frame> Add_pins_in_frame(this IEnumerable<Frame> frames)
    {
        foreach (var f in frames)
        {
            f.Score = f.Rolls[0] + f.Rolls[1];
            yield return f;
        }
    }   
    …

This is easy enough, isn´t it? A small functional unit, readily understandable, with single responsibility.

The other operations look the same. I spare you listing them here. They are all independent of each other. So they are easy to test.

But what about the composit functional units like the root? I implement them too although they are hardly doing anything. Their only purpose is to “wire up” the actions they contain. Here´s the API function for which you saw the acceptance test above:

public class BowlingGame
{
    public static int CalculateTotal(IEnumerable<int> rolls)
    {
        return rolls.ToFrames()
                    .Enrich_frames_with_score()
                    .Calc_total();
    }
}

How easy to read is this for you? How close to the model is this?

Or here the other composite action:

internal static class BowlingGameExtensions_Scoring
{
    public static IEnumerable<Frame> Enrich_frames_with_score(this IEnumerable<Frame> frames)
    {
        return frames.Add_pins_in_frame()
                     .Add_bonus_for_spares()
                     .Add_bonus_for_strikes();
    }   
    …

This too faithfully represents the model. In fact I “mechanically” translated it from the model. Each functional unit in the model either becomes an operation and is fleshed out test-first. Or it becomes a composite just plugging together other functional units in the most legible and easy way possible. There´s never a control statement in implementations of composite functional units.

There is a clear separation of concerns between operations and composites. The former need creative implementation, the latter only need mechanical implementation. They can even be generated from the model as Stefan Lieser´s solution will show.

Intermediate conclusion

Solving Ron´s problem this way was very straightforward. The hardest part was “decoding” the requirements. But once I understood the bowling score rules it took me just a couple of minutes to come up with the models.

As said above, even the nicest model diagram is no guarantee to be correct/sufficient. Nevertheless it provides a lot of value:

• I was able to talk about my mental model with my colleague.
• We were able to weigh the different approaches against each other.
• The model described the most important part of any solution: the transformation. Flow-Design models are about functionality. The put action first and structure second.
• The model provided me with small, focused, independent functional units to implement. No refactoring was necessary.
• The model sports a fundamental and important separation of concerns between “doing” (operations) and “coordination” (composites).
• The model is easy to understand on different levels of abstraction “at a glance”. (Which still requires understanding the visual notation and the domain terminology. My grandma sure would not understand these models.)
• The model is fully present in the code, i.e. each model element has a corresponding code artifact.

But this is only one possible way of translating Flow-Designs into code. It´s the easiest way, because no tooling is required and the result is readily understood. However the code lacks the capability to reproduce the model; code and model can go out of sync. Also it can become tedious to write “coordinating” code by hand.

That´s why the next article is going to show you a translation of model into Event-based Components.

PS: In case you think this solution is overengineered: Partly I agree. It´s a tiny problem that also could have easily be solved without an explicit design like this. But Ron posed this problem to challenge me, so I was required to use Flow-Design.

On the other hand you never really know. A problem might look small – but in the end, once you understand it, is not. Also this approach lead to code that´s easy to evolve since it´s already refactored from the outset. Functionality is obvious and communicatable. Reasoning about where to apply changes if necessary is easy.

TDD might have resulted in a similarly fine grained design. But that design never had been visualized so all reasoning would need to work on just code. At least I find that cumbersome.

Doing CodeKatas is all the rage lately. That´s great since widely accepted exercises are important to further the art. They provide a means of communication across platforms and allow to compare results which is part of any deliberate practice.

But CodeKatas suffer from their size. They are intentionally small, so they can be done again and again. Repetition helps to build habit and to dig deeper. Over time ever new nuances of the problem or one´s approach become visible.

On the other hand, though, their small size limits the methods, techniques, technologies that can be applied. To improve your TDD skills doing CodeKatas might be enough. But what about other skills? Developing on a software in a team, designing larger pieces of software, iteratively releasing software… all this and more is kinda hard to train using the tiny CodeKata problems.

That´s why I´d like to present here another kind of kata I call Application Kata (or just AppKata).

AppKatas are larger programming problems. They require the development of “whole” applications, i.e. not just one class or method, but bunches of classes accessible through a user interface. Also AppKata problems always are split into iterations. To get the most out of them, just look at the requirements of one iteration at a time. This way you´re closer to reality where requirements evolve in unexpected ways.

So if you´re looking for more of a challenge for your software development skills, check out these AppKatas – or invent your own.

AppKatas are platform independent like CodeKatas. Use whatever programming language and IDE you like. Also use whatever approach to software development you like. Just be sensitive to how easy it is to evolve your code across iterations. Reflect on what went well and what not. Compare your solutions with others. Or – for even more challenge – go for the “Coding Carousel” (see below).

CSV Viewer

An application to view CSV files. Sounds easy, but watch out! Requirements sometimes drastically change if the customer is happy with what you delivered.

• Iteration 1
• Iteration 2
• Iteration 3
• Iteration 4
• Iteration 5 (to come)

Questionnaire

If you like GUI programming, this AppKata might be for you. It´s about an app to let people fill out questionnaires. Also this problem might be interestin for you, if you´re into DDD.

• Iteration 1
• Iteration 2 (to come)
• Iteration 3 (to come)
• Iteration 4 (to come)

Tic Tac Toe

For developers who like game programming. Although Tic Tac Toe is a trivial game, this AppKata poses some interesting infrastructure challenges. The GUI, however, stays simple; leave any 3D ambitions at home ;-)

• Iteration 1
• Iteration 2 (to come)
• Iteration 3 (to come)
• Iteration 4 (to come)
• Iteration 5 (to come)

Coding Carousel

There are many ways you can do AppKatas. Work on them alone or in a team, pitch several devs against each other in an AppKata contest – or go around in a Coding Carousel.

For the Coding Carousel you need at least 3 dev teams (regardless of size). All teams work on the same iteration at the same time.

But here´s the trick: After each iteration the teams swap their code. Whatever they did for iteration n will be the basis for changes another team has to apply in iteration n+1.

The code is going around the teams like in a carousel.

I promise you, that´s gonna be fun! :-)

In my previous post I summarized the notation for Flow-Design (FD) diagrams. Now is the time to show you how to translate those diagrams into code. Hopefully you feel how different this is from UML. UML leaves you alone with your sequence diagram or component diagram or activity diagram. They leave it to you how to translate your elaborate design into code. Or maybe UML thinks it´s so easy no further explanations are needed? I don´t know. I just know that, as soon as people stop designing with UML and start coding, things end up to be very different from the design. And that´s bad. That degrades graphical designs to just time waste on paper (or some designer). I even believe that´s the reason why most programmers view textual source code as the only and single source of truth. Design and code usually do not match.

FD is trying to change that. It wants to make true design a first class method in every developers toolchest. For that the first prerequisite is to be able to easily translate any design into code. Mechanically, without thinking. Even a compiler could do it :-) (More of that in some other article.)

Translating to Methods

The first translation I want to show you is for small designs. When you start using FD you should translate your diagrams like this.

Functional units become methods. That´s it. An input-pin becomes a method parameter, an output-pin becomes a return value:

The above is a part. But a board can be translated likewise and calls the nested FUs in order:

In any case be sure to keep the board method clear of any and all business logic. It should not contain any control structures like if, switch, or a loop. Boards do just one thing: calling nested functional units in proper sequence.

What about multiple input-pins? Try to avoid them. Replace them with a join returning a tuple:

What about multiple output-pins? Try to avoid them. Or return a tuple. Or use out-parameters:

But as I said, this simple translation is for simple designs only.

Splits and joins are easily done with method translation:

All pretty straightforward, isn´t it.

But what about wires, named pins, entry points, explicit dependencies? I suggest you don´t use this kind of translation when your designs need these features.

Translating to methods is for small scale designs like you might do once you´re working on the implementation of a part of a larger design. Or maybe for a code kata you´re doing in your local coding dojo. Instead of doing TDD try doing FD and translate your design into methods. You´ll see that way it´s much easier to work collaboratively on designs, remember them more easily, keep them clean, and lessen the need for refactoring.

Translating to Events

Translating FD diagrams to methods does not scale well. To reap all the benefits of FD you should therefore translate designs to Event-Based Components (EBC) as they were called originally. However this term is now deprecated, since the translation does not produce components in the sense of binary units of code. Nevertheless events enter the stage to hook together functional units.

Functional units

The favored translation results in a class for every functional unit with a method for every input-pin, and an event for every output-pin:

The default names for input-pins and output-pins are Process() and Result. You might find that strange since you´re thinking of classes as “things” with many responsibilities. In FD, though, classes can be very small with just one responsibility. And this responsibility is an action.

It´s not unusual to have a functional unit called “Read lines from text file” which then is translated into a class; why then should the class have a methode called ReadLinesFromTextFile()? Process() is sufficient.

If a functional unit has more than a single input-/output-pin the methods/events surely must have different names which are denoted by pin names:

Boards and parts do not differ in their basic translation. Both become classes (or interfaces, if you like). However, parts you implement yourself in some creative way. They are the workhorses. They contain the domain logic. Boards on the other hand could be generated – or are implemented by you without much thinking. Leave aside any creativity when implementing boards.

Boards do not contain (much) code in the input-pin methods. What they are doing happens in the constructor. Their sole purpose is to connect the functional units nested within them. Thats why input-pins are just delegating the work to be done. And that´s why all nested FUs are injected into a board using ctor injection:

With FD dependencies are (primarily) used to express nesting, i.e. different levels of abstraction. So a board is dependent on its nested functional units.

Within a flow, however, functional units do not (!) depend on one another. FU A does not depend on FU B or vice versa. Also X is independent of any other FUs preceding or following it. That makes testing extremely easy (see below).

Wiring

Wires in their simplest form are just event handler assignments:

This also makes it easy to translate split/fork:

Board input-/output-pins are treated a bit differently, though. That´s because they are connected to the same kind of pins, input to input, output to output:

Join

A join unfortunately is not that simple. You should create a small standard part to accomplish the task – or you use the join class from the ebclang project at CodePlex. ebclang is an effort to provide tools to help with Flow-Design. And ebcpatterns is a sub-project collecting implementations of parts to solve recurring problems.

Here is how you´d translate a join using the Join<T0, T1> class from ebcpatterns:

Sets of data items

Events can of course fire any number of times. So there is no need to distinguish between one output packet for an input packet and several output packets. Take as an example a functional unit splitting text lines into words:

One word or many… that does not make a difference for the translation of the output-pin to event Action<string>.

But there is a difficulty for any receiver of the output packets. Which word is the last word in a line? If several lines are processed and if it makes a difference to which line a word belongs, then a receiver has no way of associating a word with a line. The only way would be to send a special End-of-Line word as the last word of every line.

Fortunately there is another way of designing this. Just make it clear that output packets consist of several entries:

The star after the typename signifies the packet to be a list of data items. And a list is most simply represented by an IEmumerable<>.

Please note: This also works if the list contains millions of entries. Just don´t create an array and pass it along, but use an iterator (yield return) instead.

Explicit dependencies

Explicit dependencies are translated in a very explicit way. Instead of injecting them into the ctor an interface is used:

This has to advantages:

1. Injection does not interfere with any other part of the implementation. It does not force a ctor or an additional parameter to the ctor, it does not require a base class.
2. Injection is independent of object creation; injection can take place at any time during start-up of a Flow-Oriented application.

Entry Point

The entry point attribute is likewise translated to the implementation of an interface:

Configurable

Also making a part configurable is translated to the implementation of an interface:

Testing

Testing of functional units is easy:

With board you just do integration tests. Check only if the wiring is correct. Every path through a flow needs to be tested only once. Once you start using tools to generate boards these integration tests are not needed anymore.

With parts do unit tests as usual. Note that you don´t need a mock framework for that anymore because there are no dependencies between parts. Just pay attention to how to check the output:

You need to assign any relevant output event handlers before you call an input-pin method of the part. And don´t do the assert in the event handler because the test would go green even if the handler is not called.

Hosting – Putting it all together

Hosting the code created by this translation usually follows a pattern. It runs through a couple of phases:

1. Build: Create all instances of functional units
2. Bind: Wire-up the FUs by connecting output- to input-pins
3. Inject: Inject explicit dependencies on all FUs implementing IDependsOn<T>
4. Configure: Pass the command line args to all parts implementing IConfigurable
5. Run: Call the Run() method with the command line args on the sole part implementing IEntryPoint

Build and Bind are put in a sequence to distinguish them; in reality, though, they are intertwined since binding happends also in ctors of boards upon creation.