IODA Architecture by Example

The IODA Architecture to me is fundamental. But as many fundamental notions it might be a bit abstract. That's why I want to provide an example of how to design and implement software based on it. I keep things easy, but not trivial. So here is a scenario:

Build an application that translates a roman number entered by the user into an arabic number and vice versa.

This sounds easy, doesn't it. It's more than the code kata Roman Numerals requires, but not much.

Requirements analysis

Before I start with the design, a short analysis seems to be in order.

The requirements ask for an application. That means the user should be able to start a program which converts numbers. But how should the user interact with the program? Since the requirements don't tell me much about it I assume the simplest approach: a console application. It could work like this:

$ convertroman XIV
14
$ convertroman 42
XLII
$

Roman number conversion is no rocket science. I won't go into it here. Look it up on Wikipedia, if you like.

What about validation? How should the application react to negative arabic numbers or invalid characters in a roman number? The requirements are silent about this. So I assume just very, very basic validation.

The main purpose of this analysis is clarification of how data gets into the program and how output is presented to the user.

Solution design

The IODA Architecture does not prescribe any behavioral responsibilities. So what can I start with? I could dig into the requirements and identify candidate classes, if I was doing traditional object orientation. But I'm not.

Objects - at least the traditional ones - are secondary. What is primary is behavior. What's supposed to happen? That's the driving question.

At the beginning I know very little. I just know, "everything" needs to happen upon a trigger from the user.

image

But then, when I think about it, I do know a bit already. It's just a pattern:

  1. The input data has to be read. It's passed in via the command line.
  2. The data has to be converted.
  3. The conversion result has to be displayed.

Flow Design

The overall behavior is made up of "parts"; there are distinct processing steps. It can be refined. The current functional unit "convert roman" becomes the root of an integration hierarchy.

image

Now I repeat this: For every functional unit I ask myself, from what even smaller processing steps it could be made up of. If I truly understand the problem I'm sure I'll be able to name at least a few such sub-tasks.

  • How complicated is it to read the input number from the command line? It's trivial. No refinement is needed.
  • How much effort is it to display the result? It's trivial, too. No refinement needed.
  • How does the conversion work? That's not trivial. Refinement seems necessary.

Conversion has at least two sides to it: converting a roman number to its arabic equivalent, and converting an arabic number to its roman equivalent.

Also which conversion to choose needs to be decided.

And how about the simple validation?

Let's put these behavioral pieces together. Here's the "domain process" to do the conversion:

image

In the former design "convert" was a leaf, an operation. But now it has been opened up and become an integration.

Again I repeat asking myself for every functional unit how difficult it seems to implement it. If I clearly see an implementation before my mental eye, I won't refine it further during design. Otherwise I need to do a bit more thinking. How could the partial behavior possibly be accomplished as a "domain process"?

To me such a refinement seems to be warranted for both conversions. They are at the heart of the application. That's really the core domain of the program.

Here is the exemplary decomposition of "convert from roman": My approach to solve the problem consists of three processing steps.

  • Each roman digit of the roman number is translated into its value, e.g. "XVI" -> [10, 5, 1].
  • The values are summed to calculate the arabic number, e.g. [10, 5, 1] -> 16.
  • In case of a smaller value standing in front of a larger value the smaller value is negated, e.g. [10, 1, 5] -> [10, -1, 5]. I call this "applying the subtraction rule".

Correctly wired together the "domain process" consisting of these three steps looks like this:

image

If you, like try the same for the conversion of arabic to roman numbers ;-)

Should the new functional units be refined further? I don't think so. I feel confident to just write down the code. It's pretty obvious, isn't it?

So much for the happy day case. But what about a rainy day with invalid input? What to do about it? I think some error output would be nice. I'll extend the data flows from above like this:

image

It's never too late to improve a design ;-)

Let's step back: What is this I've designed so far? It's data flows; I call it Flow Design. It's a declarative description of what's supposed to happen to solve a problem (which always is some transformation of input into output and side effects).

At this point I don't have a clue how the actual logic in each of the operations will look like. That's the imperative part. It's implementation details.

But my feeling is ;-), if all operations are implemented correctly, then the integration of them in the depicted manner will lead to a correctly working program.

Here's the complete final Flow Design for the program with operations marked with dark color:

image

Data Design

So much for the I(ntegration) and O(peration) strata of the IODA Architecture. But what about the D as in Data?

Of course it's already there. Data is flowing between the operations. Here's a section of the Flow Design across levels of abstraction with focus on the data:

image

Due to the simplicity of the scenario the data types are all primitive. It's strings and integers. Maybe, yes, maybe it would be nice to introduce specific types for domain concepts like roman number and arabic number. But right now I don't see much value in that; it would feel kinda artificial/overengineered. Especially since I'm going to use C# as the implementation language. New types are comparatively expensive using that syntax; I'd be more inclined to define types like that if I was using F#.

Anyway, whether the scenario calls for its own data types or not, this does not change the basic architecture: operations depend/work on data. If there were any specific data structures they'd reside beneath the operation stratum.

Class Design

The "bubbles" you've seen so far will be translated to functions in a systematic way. Wait for the implementation...

Functions are modules. And I think each of the bubbles designed so far has a pretty narrow responsibility. It focuses on a single aspect of the requirements. See my article on the Single Responsibility Principle (SRP) for details.

But as important as it is to focus the responsibility of each function (aka bubble), there are more module levels available. I should use them to form higher level responsibilities.

Also, any functions that might be created from the bubbles designed need a class as its home.

Before moving on to the implementation I need some clarity about what classes the program should consist of.

Notice the difference between the usual OOD/P approach. I haven't thought about classes first, but about behavior/functions. Behavior is primary, classes are secondary.

So I did not have to come up with any "candidate classes". My classes are the real ones right away. And I know exactly which functions go where. Because I abstract classes from patterns I see in the Flow Design.

Here's my idea of how to group bubbles into classes:

image

Some of it might be obvious for you, some not. So let me explain my thinking:

  • Providers: Using an API is a "hard" aspect in any software. It should be isolated into specific modules. Usually I'd say each API should be encapsulated by its own class - but in this scenario there is so little of it, I guess a single class will do.
  • FromRomanConversion and ToRomanConversion: Doing the actual conversion is an aspect of its own. Since there are two directions of conversion a module for each seems to be in order.
  • RomanConversions: This class belongs to the coarse grained domain aspect of the application. It contains "helper functions" not actually concerned with the transformations.
  • Body: This class represents the overall functionality - but without the interaction with the user. It's like an internal API to what the program is supposed to do.1
  • Head: The head is responsible for triggering body behavior. It integrates the body with input from and output to the user.

Interestingly the class diagram follows the IODA Architecture like the bubbly Flow Design:

image

Only the data stratum is missing. But that's just because the scenario is so simple.

Interfaces and Static Classes

Class design is more than determining which classes there are. It's also about how the services of those classes should be accessible.

Question #1: Can any classes be static? I'd say that's a legitimate question; not all classes need to be instantiated. This only makes sense for classes with data or those which should be replaceable or into which replacements (mocks etc.) should be injected.

Looking at the class design for this application I think, RomanConversions, ToRomanConversion, and FromRomanConversion can be static. There is no state involved, no APIs need to be replaced for testing purposes.

This is different for Providers. It encapsulates APIs which might make it hard to test its users like Head. An interface seems in order or maybe even two: one for the CLI part and one for the console output part of the class. Why not apply the Interface Segregation Principle (ISP)?

From that follows Head cannot be made static. It needs to get instances of the provider interfaces injected, since it statically depends on the interfaces and dynamically on some implementation of them.

Also Body should be described by an interface. That would make it easy to test the head in isolation.

Library Design

Aspects are hierarchical so they should be mapped to a hierarchy of modules. Functional units of the Flow Design have been aggregated into classes. But what about the classes?

The next level of modules after classes is libraries. At least in my thinking.

Also libraries are a means of organizing code in many IDEs under the name "project" (eg. Visual Studio) or "module" (eg. IntelliJ).

Again I don't start with any library candidates, but rather let the classes inspire me. What should be grouped into a library? Here's my answer:

image

  • RomanConversions contains all classes of the business domain.
  • Providers is the home for all classes dealing with APIs.
  • Head is the top level integrator, the entry point of the application.
  • Body does the integration of the "backend". That's where behavior is created.

Component Design

The next level in the module hierarchy are components. A component consists of one or more libraries and is described by a separate platform specific contract. This enables a more "industrial" style of software production, since work on different components can be done in parallel, even if they depend on each other at runtime.

The current scenario does not really call for components. I'm working on it alone. But still... I'll make Providers and Body components, since there are interfaces and possibly several implementations over time. It's not much effort, so there will be another library for the contracts.

Implementation

Implementing the design is easy. Almost boring ;-) There are clear rules how functional units of Flow Designs should be translated into functions. It took me some 90 minutes to write down the code including all tests.

image

Test-first coding wasn't necessary since all operation functions are so small. Also the design was already there so I did not need tests to drive it. The tests I set up are there to avoid regressions.

You can view the sources on GitHub.

When looking at the code there should not be any surprises. The project layout follows the component/library design.

image

The dependencies between the libraries/classes are as planned.

image

The code has a IODA architecture on all module levels from functions to components.2

One thing, however, is noteworthy, I guess. How did I translate the bifurcations in the Flow Design, i.e. where the flow splits in two? There are two outputs to "convert", "determine number type", and the validations.

When you look at the following code you might need a minute to understand what's happening:

image

But actually it's simple. Just read the function calls from top to bottom and left to right. Indentations denote different paths in the flow.

Here's the topmost flow side by side with its implementation:

image

Each bubble translates to a function call. If there is only one output, the function returns a value. If there is more than one output, the function passes on a value using a continuation (function pointer). Those continuations are what makes the code a bit hard to read. But only at first. It's unusual, not wrong. You just have been trained for so long to read nested function calls from right to left and inside-out; but once you get used to continuations you'll come to appreciate how easy it is again to read code.

Elements of Functional Programming are used to translate the declarative Flow Design into OO language code.

But that's only details. My main point was to show you, that designing a IODA Architecture for an application is not difficult. And implementing the design is not difficult either.

The universal aspects of integration, operation, and data (admittedly not much in this case) have been separated. And orthogonal to that behavioral aspects have been identified and encapsulated into modules on different levels.


  1. This root class could also be called Application or maybe viewed as a use case and named accordingly.

  2. There are slight exceptions. Although the classes FromRomanConversion and ToRomanConversion are operations from the point of view of the class design, they contain both integration functions and operation functions. As long as there is a large imbalance between the two aspects that’s ok. If the number of integration functions would grow in such a hybrid class, though, it would hint at splitting it into dedicated integration and operation classes.

The IODA Architecture

The common architectural patterns are all pretty much the same. Layered architecture, MVC, Hexagonal architecture, Onion Architecture, Clean architecture… they all do two things:

  • Define domains of responsibility
  • Put functional dependencies in order

Look at these depictions of some of the patterns:

image

What you find are shapes (rectangles, circles) with the responsibility to produce part of the required behavior of a software. All models forsee certain responsibilities like interaction with the user, persisting data, representing and working with domain data etc. Some patterns are more detailed, some are less. That way they are more or less applicable to concrete requirements.

And those shapes are explicitly or implicitly connected in service relationships. One shape uses another; one shape depends on another. Sometimes arrows denote these relationships, sometimes mere location (e.g. a shape on top of another means “top uses bottom”).

These uses-relationships are cleanly directed and there are no cycles.

This is all nice and well. It helped us along quite a while.

But again and again I’ve seen developers scratching their heads. What about stored procedures in a database? They implement business logic, but don’t reside in the Business Layer. What about JavaScript validation logic? It’s part of the Presentation Layer, but implements business logic. Or what if I don’t want to go down the DDD path with Domain Entities and Services? Maybe the application is too small for this.

Most of all, though, what boggles the mind of developers, as far as I can see, are the functional dependencies between all those responsibilities. It’s deep hierarchies which won’t go away by switching to IoC and DI.

Looking at a function on any level of this hierarchy is an exercise in archaeology. Put on your head lamp, get out the brush, be careful while you uncover mysterious code hieroglyphs of the past. There is so much to learn from those 5,000 lines of code… And there, look, another tunnel leading you to yet another level of code.

As useful the architectural patterns are, they are only coarse grained and they leave us with a very crude idea of the fundamental structure of software. According to them, software is just a deep hierarchy of services with particular responsibilities calling other services calling other services etc. for help.

Thus the DI container has become the developers best friend. And his second best friend has become the mock framework.

This is bothering me.

Certainly every software system is made up of aspects which should be encapsulated in modules. There might be a UI aspect or a persistence aspect or a use case aspect or an encryption aspect or a reporting aspect etc. etc. They are all a matter of the Single Responsibility Principle, which I’ve already put under the microscope.

At this point I don’t want to argue with the established architectural patterns about the responsibilities they perceive as inevitable, be that Business Layer or Repository or Controller etc.

Let’s call them behavioral responsibilities. Because each responsibility is about a functional or non-functional aspect of the behavior the customer wants a software to show.

But what I question is, whether the fundamental notion of software being just a uniform hierarchy of services is useful. And service being some kind of behavioral responsibility to be used by another service.

image

Essentially there is no distinction. Look at the code in a function in a Presentation Layer or a Use Case or a Controller or a Respository. You won’t see a difference - except in behavioral purpose. All functions on all levels of the dependency tree contain logic and calls to other functions.

“That’s how it is. That’s the natur of software, isn’t it?”, you might say. But I beg to differ. This is just how we structure software today. It’s a mush. And that’s why there are functions with 5,000 or even 10,000 lines of code. There is no technical limit to how many LOC a function can have. And there is no artificial rule about how many LOC it should have (except for some recommendations). It’s all up to us developers. So if push comes to shove a method just grows - one line at a time. Because it can.

This is what has been happening for the past 50 years. And it has been happening despite all well meant design patterns or architecture patterns.

Why’s that? My guess, it’s because all those patterns do not question the basic assumption of all levels of dependency being created equal.

Patterns so far have been concerned with defining certain behavioral responsibilities and nicely ordered dependencies between them. That’s all.

So I suggest, if we don’t like the current state of codebase affairs, then we should try something new. Here’s what:

Formal responsibilities

I’d like to propose to search for the fundamental architecture of software in another dimension. The current architectural patterns are just evolutionary steps in the same dimension. So let’s step out of it. Let’s turn 90° and start moving down another path.

It’s one thing to search for responsibilities in the behavioral domain. That’s necessary, but also it is limited. Like the Flatlanders baffled by the effects of a 3D object passing through their 2D space, we’re again and agin baffled by the temptation to name some class “manager” or “coordinator”. It doesn’t sound right.

Locked in our current view of software design the need for managercoordinatorcontroller classes seems like a symptom of bad object-orientation.

But what if we broaden our view, step out of our dimension or behavioral responsibilities?

Let me introduce three formal responsibilities. I call them formal, because they are not concerned with creating a certain behavior. These formal responsibilities are orthogonal to all behavioral responsibilities.

Operation

Operation I call the responsibility of any logic. (With logic being transformations/expressions, control statements, hardware access/API calls.)

Any module containing just logic, is an operation. What an operation does, whether it stores data in a file, calculates a price, parses a string, stuffs data into a view is of no concern.

It’s implicit in that operations only contain data. However, since this is so important, let me state it also explicitly: Operations may not call any other operation. Operations don’t know of each other. There is no functional dependency between operations. Calling a function is not part of the definition of logic (see above).

Operations just execute their own logic which works on data. They are IO-processors: given some input they produce output (and possibly side effects).

Data

Giving structure to data is a responsibility separate from operating on data. Operations of course work on data - but they are not data.

This might sound opposed to object-orientation, but it is not. Data structures may provide services to work with them. Those services just should be limited to maintaining structure and consistency of the data.

If something is data, then that’s what its functionality should be limited to. If, on the other hand, something has data, it’s free to operate on it in any way.

Integration

Since operations don’t know each other, there needs to be some other party to form a visible behavior from all those small operational behaviors. That’s what I call integration.

Integration is the responsibility to put pieces together. Nothing more, nothing less. That means, integration does not perform any logic. From this also follows, integration is not functionally dependent on operations or other integration.

Formal dependencies

The formal responsibilities (or aspects) are independent of any domain. They neither suggest nor deny there should exist adapters or controllers or use cases or business logic modules in a software. They are also independent of any requirements scenario. Software can (and should) be structured according to the formal aspects if it’s a game, a web application, a batch processor, a desktop application or a mobile application or some service running on a device.

The formal aspects are truly universal. They define fundamental formal responsibilities to be separated into modules of their own. Put operations into other modules than integration or data.

And interestingly what goes away if you do this are functional dependencies between the modules of an application.

If you decide to have some operation module for displaying data, and some for doing business calculations, and some for persisting data… then those modules won’t be calling each other.

image

They just float around independently. Only maybe sharing some data.

image

As you can see, operations depend on data. Of course that’s necessary. Otherwise there would be no purpose for software. Logic needs to have raw material to process.

But data is not functionality. It’s, well, data. It’s structure.

Finally those operations floating around need to be wired-up to work in cooperation towards a common goal: some larger behavior of which each is just a part.

This is done by integration on possibly many levels.

image

Integration depends on operations and other integration. But also this dependency is not “bad”, because it’s no functional dependency.

As you can see, dependencies won’t go away. We can’t build software without them. But the functional dependencies of the behavioral architecture patterns are gone. There are no more services calling other services etc.

The dependencies here are just formal. Or empty, if you will. They don’t have to do with any functional or non-functional efficiency requirements.

Separating What from How

I call this universal separation of formal responsibilities IODA Architecture. There are several strata of Integration sitting on top of one stratum of Operations using Data - and APIs from outside the scope of the requirements to at least cause any tangible effects on hardware.

image

Or if your more inclined towards “onions” here’s another one ;-)

image

Or this one, depending on how you want to look at hit. You could say, the environment interacts with some integration, so they need to be on the surface. Or you could say, the environment is only accessible through APIs, so they need to be on the surface.

image

In any case, operations are about how behavior is created. Logic is imperative, it’s nitty gritty details.

Integration on the other hand defines what is supposed to happen. It does not contain logic, it thus is not imperative, but declarative. Like SQL is declarative. Integration assumes the stuff it integrates just to work. If that assumption is valid, then it promises to wire it up into something larger and also correctly functioning.

Also data is about the what. No logic in data except to enforce a certain structure.

Self-similarity

I’ve long sought an image to symbolize the IODA architecture. And now I’ve found one which I like quite a bit: a Sierpinski triangle.

image

I think it fits because it’s a self-similar figure. It’s a triangle made up of triangles made up of triangles… It’s triangles all the way down :-) A fractal.

IODA is also self-similar: An operation on one level of abstraction can in fact be an IODA structure - which you only see when you zoom in.

image

 

image

Operations on a high level of abstraction are black boxes. They are leaves of the the behavioral tree - which works on data and uses APIs.

But if you think, an operation is too coarse grained, you may open it up at any time. Refine it. Zoom in. Then it’s decomposed into another “miniature” IODA hierarchy. The integration formerly integrating the operation with others then integrates the root integration of the operation.[1]

Summary

To escape from “dependency hell” it’s not enough to wave the IoC/DI wand. The problem is more fundamental. It’s the very assumption software should be built at deep hierarchies of functionally dependent modules that’s problematic.

Functional dependencies lead to ever growing methods. The SRP is too weak to keep developers from adding more lines to an already excessively long method.

This is different in a IODA architecture.[2] There simply is no way to write an operational method with more a dozen lines of code. Because any developer then will tend to extract some logic into a method to be called. But that would be a violation of the IODA rule of not calling any (non-API) functions from an operation. So if a method is extracted from an operation’s logic the operation has to be turned into an integration.

Integration methods are also short. Without any logic being allowed in an integration method it simply does not feel right to have more than a dozen lines of code in it. And it’s easy to extract an integration method if the number of lines grows.

IODA does not only get rid of functional dependencies - of which a palpable effect is, the need for mock frameworks drastically diminishes -, it also creates a force keeping module sizes small.

Software architecture is not a matter of whether a Presentation Layer may only call a Business Layer or Use Cases may call Entities or the other way around. There should not be any service dependencies between these or other kinds of behavioral responsibilities in the first place.

Which behavioral responsibilities there should be is only a function of the concrete requirements. But what is truly universal and unchanging in my view is the distinction between integration, operation, data - and whatever framework APIs deemed helpful to get the job done.

image


  1. The same is true for data, by the way. Data structures are black boxes to operations. But since they can contain logic (to enforce their structure and consistency), they might not just consist of operations but a whole IODA hierarchy.

  2. Which is not just a theoretical thought, but experience from several years of designing software based on IODA.

Sweet Aspects

When a module has a single responsibility that means it’s focused on producing the behavior of a single aspect. That’s how I described the Single Responsibility Principle (SRP) in my previous article.

Since the SRP is so important and at the same time a bit elusive, I thought, I try to illustrate this “traits and aspects thing” a bit more. Maybe you can even develop a taste for the SRP :-)

Look at this heap of sweets:

image

Let them represent logic in your code. You know, those 5,000 lines of code in a method, or those 200 methods in a class, or those 1000 classes in a library… It’s all the same. Unstructured stuff. High entropy. Disorder. Not clean. Yet at least ;-)

At some point it might have been a good thing to throw all this together. Maybe that way delivery was faster/easier? You know all those explanation how legacy code just happened over time. Should get into technical debt very consciously; have a plan how to pay it back. But in my experience that’s not how it works for teams. Like many private households (and companies) they stumble and falter and fall; or each time they they borrow it’s an exception and sure will never happen again. Promise! Until the next emergency which requires an exception.

5,000 line methods happen. They happen one line at a time. And each time there is a very good reason why to add the line instead of first refactor the whole heap of logic.

So that’s what there is at the end for 5,000 very good reasons: a huge bag of stuff. A brownfield.

image

But now is the time to bring order to the chaos ;-) Refactoring time. Let’s apply the SRP. It will make it easier to get an overview of what’s there, to make it easier to pick just the sweet you like most. Or in code: Order will make it easier to understand code, when you want to add new features or fix a bug. Also extending the code will be smoother. Less refactoring, less digging around, less unwanted side effects.

But what kind of order should there be? How to group the parts? Which traits to select to define sets with high cohesion between their elements?

I think, with regard to the depicted sweets people are divided into those who like licorice and those who don’t ;-) So “taste” could be a single trait to split the sweets into sets.

One set has the responsibility to make people happy who like licorice. The other one has the responsibility to make those happy who don’t like it. And then there is a third set with just one sweet which tastes like a gummibear, but with licorice.

image

Picking just a single trait splits a whole into as many parts as there are values the trait can assume. That might lead to a coarse grained grouping aka modularization as you can see above.

So now let’s pick two traits and see what happens, e.g. “taste” and “form”:

image

You see, this doubles the groups! There is one for (licorice, cube), (licorice, tube), (gummibear, bear), (gummibear, fruit), (gummibear, frog), (licoricegummibear, bat).

And again. Now with yet another trait: “taste”, “form”, “color”.

image

Wow! More than 20 groups of sweets. A fine grained modularization, I’d say.

The closer you look, the more traits you learn to distinguish, the smaller the groups/modules covering an aspect become. Yes, that what I’d call the selected traits: an aspect.

The combination of taste+form+color we could call the “gourmet aspect” ;-) It’s relevant for someone who relishes these kinds of sweets.

But someone who needs to pack them up (or is very hungry) might be more interested in the “size aspect” which groups according to the trait volume.

image

Or someone who wants do use sweets to lay out a picture with them might just look at the “color aspect”:

image

So as you can see, cohesion sometimes is tangible, but sometimes it’s not. For sweets color, form, size are “hard” traits. You don’t need experience with “using” the sweets in order to group them along these lines. In software it’s easily visible traits like APIs, data, logic that lead to “hard aspects”. You don’t need to understand the domain for modularization - but a scrutinizing view helps.

The trait taste is different though. You need to experience the sweets. Only then you can distinguish values and group the sweets. That’s comparable to parts of logic where just looking at it won’t help to find aspects. You have to really understand it’s parts. What’s the purpose of each line, of groups of statements? Only then you realize their “taste” and can separate them into focused modules.

The result: What formerly was just a heap of stuff - sweets or logic - now is a nicely structured, clean, modularized system of code. Each module having a clear single responsibility on one of several levels of abstraction.

image

I hope this excursion into an analogy helps you to get am even better grip on the SRP. It’s worth trying. Good modularization is at the heart of any clean coding effort.

Bon appétit! :-)

The Single Responsibility Principle under the microscope

It’s equally important and mysterious: the Single Responsibility Principle (SRP). And even though its originator Robert C. Martin tried to explain it again, I feel there is something lacking. Some concreteness, some tangibility.

First the definition of the SRP in Martin’s own words:

“The Single Responsibility Principle (SRP) states that each software module should have one and only one reason to change.”

This is more informative than Wikipedia’s tautology:

“In object-oriented programming, the single responsibility principle states that every class should have responsibility over a single part of the functionality provided by the software, and that responsibility should be entirely encapsulated by the class.”

But still… Martin’s definition is not crystal clear. It leaves much room for interpretation during TDD refactoring or code reviews. Even he admits this:

“However it begs the question: What defines a reason to change?”

To improve on this, he developed the notion of a “responsibility owner” as I’d call it:

“When you write a software module, you want to make sure that when changes are requested, those changes can only originate from a single person, or rather, a single tightly coupled group of people representing a single narrowly defined business function.”

That’s a clever idea - but it’s only useful for pretty coarse grained responsibilities. Take an online shop for example. I assume we can agree on such a shop having a couple of distinct parts like catalog, shopping cart, recommendation engine, order processing, billing etc.

image

For each of those parts there probably is a person (stakeholder) owning it, i.e. having a clear vision of how this part of the shop should behave and look like. According to Robert C. Martin this would mean, I guess, each functionality should be put into a separate module.

Who would disagree?

But that’s not really a difficult decision. Yet.

Now, what about a small company ordering such an online shop. There is only a single person responsible for all the parts. Would that mean everything has to go into just one module? Certainly not. Because what Robert C. Martin probably means it not person, but role.

Even if there is only a single responsible person for so many different parts of the software, this person views the whole from different perspectives. Mentally she switches between different roles like accounting, sales, marketing, support, even customer.

Does that help defining what a “reason to change” is? The SRP could be rephrased like this:

Separate code into modules so that each module is owned by only a single business role.

This is not bad. But still it breaks down once you zoom in on such a module. How to apply the SRP within (!) the catalog module? The whole catalog is owned by the sales role. But since it does not seem realistic to implement the whole catalog in a single function or even class, there is no guidance from the owning role as to how to form smaller modules.

I’m sorry, but even this augmented definition of the SRP does not seem to help much. My guess is, we need to look at the matter a bit more systematically.

Requirements

First let’s be clear about what responsibilities are: If a module has a single or even multiple responsibilities, then that means it contributes to the fulfillment of requirements. A module can be responsible for some functional requirement and/or some non-functional efficiency requirement (e.g. performance, scalability).

Contribution to the fulfillment of such requirements means, the logic in a module produces a part of the desired behavior. Because that’s what software is all about: behavior. Customers want software to behave in a certain way, e.g. show the products in a certain category when clicking on an image (functional requirement) or display a query result in less than 1 second (efficiency requirement).

Behavior is triggered by some input from a user. It consists of output to the user and/or side effects (e.g. changes to a database or printing a document).

imageBehavior is produced solely by logic, which to me are transformational statements (expressions), control statements (e.g. if, switch, for, repeat), and API calls/hardware access.

That’s it. That’s what customers want. Nothing more, nothing less. Which means, they don’t want modules. Not a single one.

Modules

To wrap logic up into modules does not contribute to the behavior of software. So why do it? Why should we wreck our brains about whether a module has already just a single responsibility or not?

We need modules for the non-functional requirements of evolvability and productivity. That’s their sole purpose. Modules don’t contribute to functionality or efficiency; they only make it easier to keep adding features to the code and fixing bugs.

Customers want that, too. Unfortunately they take it for granted. And we developers have a hard time to explain why keeping evolvability high is not that easy. Especially because there is no hard-and-fast metric for evolvability.[1]

So there are now orthogonal responsibilities: the responsibility with regard to functional and efficiency requirements - I call that logic responsibility -, and the responsibility with regard to evolvability.

Following the SRP thus means to identify some logic responsibility and wrap it up into a module to fulfill the evolvability responsibility.

image

Hierarchies

From the above example it should be obvious there is not just a single level of responsibilities. Responsibilities can be coarse grained or fine grained.

A module representing the catalog functionality of an online shop has a pretty broad, but still single responsibility.

But this functionality consists of different sub-functionalities and efficiencies. I imagine there to be maybe a query engine and some price calculation and a cache and several other parts to play together in order to give users a “catalog experience”. And although they are all owned by one role they should not go into the same module, I’d say.

That means a hierarchy of requirements has to be mapped onto a hierarchy of modules. But should those be all of the same kind, e.g. classes? That would dilute the hierarchy. It would not be visible in the code base anymore, at least not that easily. There would be just a bunch of classes (for query engine, calculation etc.), of which maybe one would be conceptually on a higher level (for the overall catalog).

That would work, but I suggest we broaden our view of modules. Let’s think of modules forming a hierarchy of their own.

In his first version of the SRP Robert C. Martin spoke of classes. In the updated version he speaks of modules - but leaves it to the reader to figure out what modules are. I think that doesn’t make it easier to understand what the SRP means.

So here is my take on what modules are:

Modules are design time code containers to enable evolvability and productivity.

This might sound a bit lofty, but in fact it’s pretty simple. To make it more concrete here is my hierarchy of modules from fine grained to coarse grained:

  1. Functions
  2. Classes
  3. Libraries
  4. Components
  5. (µ)Services

image

A level n module can be said to physically contain level one or more n–1 modules. Of course each level has its own characteristics. It brings something special to the table: Functions abstract logic behind a name, classes encapsulate details about data and functions, libraries are true black boxes, components separate contract from implementation, and (µ)Services sport platform neutral contracts.

Now think of it: The sole purpose of all these containers is to make it easier for your to maintain your code. They are used to encapsulate details, they save time by letting you re-use them, they decouple through contracts…

None of these containers was invented to improve functionality or performance or scalability or security.[2]

Even µServices, the newest kind of module, are not a matter of scalability or security. They are supposed to improve evolvability at design time and runtime and make polyglot development easier.[3]

With such a physical hierarchy of modules it’s easier to map a hierarchy of required behavior to code. For example the catalog could become a (µ)Service. Then the query engine and the calculation could become components within this (µ)Service. And the even smaller behavioral parts could become libraries and classes and functions. All nested within each other like russian dolls.

With a physical hierarchy of modules this can be done. It helps to map the reality of hierarchical requirements. But of course even this comes to an end. There might be more behavioral levels in the requirements than levels of modules. So in the end modules of the same level need to be arranged in some kind of logical hierarchy.

Anyway, each module on each level is supposed to have a single responsibility. But if there are maybe 5 components in a software system with 3 libraries each with 10 classes with 10 functions each… that would amount to 1,500 modules. And I don’t think you can or want to find 1,500 persons (or roles) out there who own these modules.

Such a simple calculation makes it obvious: The notion of “responsibility equals owner” is only helpful for very coarse grained modules. Don’t neglect it, but don’t put all your responsibility eggs in this basket.

Aspects

The field the SRP targets now has more structure, it has become more detailed. But my feeling is, we’re still close to square one. Robert C. Martin’s question has not been answered yet:

“What defines a reason to change?”

Which logic to put into this function and not into that one? Which functions to split among two classes instead of keeping them in just one? Which classes to group in a library? Why split a libary into three? etc. That’s the questions we have to ask during design/refactoring all the time.

Let me try to move closer to an answer by introducing the notion of an aspect. The term is not new to software development, but I want to use it in a more general way than AOP (Aspect Oriented Programming) does.

Every module has a purpose. In some way it contributes to the overall behavior of a software. The more it focuses on a certain aspect of the behavior, the better. Then it has a clear responsibility.

A module’s responsibility is to implement an aspect of requirements.

An aspect can be functional or non-functional. Aspects form a hierarchy, they can be on different levels of abstraction.

The above mentioned catalog, shopping cart etc. are aspects of the overall solution. They are easy to identify, but below that…? They are just examples. So what’s the definition of aspect? Mine is:

An aspect is a set of traits on a common level of abstraction which are likely to change together.

This might sound complicated, but in fact we deal with aspects every day. Think of a person. What are aspects of a person?

I’d say for example hairdo and clothes are aspects of a person. Each stands for a set of traits, and each trait can take on one of several values, e.g.

  • Hairdo: hair color (black, blond, red…), hair length (long, short, bald…), hair style (curly, lank, pony tail…) and more.
  • Clothes: color (black, brown, purple…), style (casual, festive, rugged…), fabric (denim, silk, linnen…) and more.

image

What binds traits together to form an aspect, what makes up their high cohesion is how they change. We view traits as cohesive when they are affected by the same causes which means they tend to change at the same time.

It’s more likely to change the hair length at the same time as the hair color and the hair style. This happens when you go to a hair dresser. The synchronicity is high, the frequency every couple of weeks. The cause might be you want to look nicer for a party or because bringing your hair in order every morning has become exhausting ;-)

It’s also more likely to change the color of your clothes at the same time as the style and the fabric. This happens when you dress yourself in the morning or dress up to go to the opera. The synchronicity is high, the frequency a couple of times per day. The cause might be you want to feel more comfortable or more appropriate for the occasion.

On the other hand it’s less likely to change your hair style and at the same time also change the color of your clothes. At least it’s that way for the people I know ;-) Which means: If there are circumstances which lead to changing hair style and clothes color more or less together, if there is a common cause for both, then… well, then it’s probably useful to view them as an aspect.[4]

image

This kind of abstraction we do every day. There are thousands of already agreed upon common aspects.

But when you are confronted with software requirements… then there are no aspects at first. It’s just a vast land of traits. It’s tons of details without abstractions. That’s why the SRP is at the core of our profession. We’re aspect hunters.

Here’s the analogy again:

First there is just a bunch of traits: hair color, clothes color, hair length, fabric etc.

Then we see those traits change their values: hair color goes from black to blond, hair length goes from long to short - but clothes color stays the same.

Then we realize there are patterns in those changes.

Then we abstract from the traits. We group them together according to the observed (or imagined) change patterns (which represent causes). Aspects are born.

In a myriad of real (or anticipated) changes to traits we need to spot patterns and then abstract. Aspect hunters are masters of abstraction. That’s what the SRP is about.[5]

Sure, ultimately all changes originate from people. That’s why Robert C. Martin says:

“[The SRP] is about people.”

But this is only helpful at the beginning. It’s a shortcut to abstraction. We don’t want to wait for changes to happen before we venture into abstraction. We want the abstractions to be served on a silver platter.

There is nothing wrong with shortcuts. But we should recognize them as such - an be prepared to go beyond them.

Hunting for aspects top-down

Spotting cohesion is no easy feat. It becomes easier with experience, though. Fortunately. But still, it’s tough. So we should look for help.

Robert C. Martin provides help when he recommends to look to stakeholders and their roles to find aspects that are worth to be represented by a module. Such roles as owners of a coarse grained aspect mostly help to structure the domain of a software into coarse grained modules.

Take the online shop for example again. There will be a person most interested in the presentation of the products, and a person responsible for the pricing, and a person responsible for order processing, and a person for billing etc. What should you make of them?

Since that’s pretty coarse grained aspects of the problem domain I find it natural to derive from this high level modules, e.g. (µ)Services or components. A catalog service or pricing component would make sense to me, but probably not a class to represent the whole aspect of price calculation.

This is an example of finding aspects top-down. You start with a business role as the root and drill down into to find requirements aspects.

When you look at patterns like the layered architecture or Ports-and-Adapters or Clean Architecture you find more examples of top-down aspect determination.

image

These patterns are congealed experience telling us: There are aspects common to all software regardless of the domain. Let’s call them cross-cutting aspects as opposed to domain aspects.

For example, interacting with the user is an aspect to be separated from persisting data or communicating with other software. These should be wrapped into modules. But of what size, on what level of the module hierarchy?

I think, such aspects should be encapsulated by at least a library; to view them as black boxes helps testing and exchanging them to keep up with technological development. These aspects form the interface of a software.

But there is more: Robert C. Martin zooms in on what’s hidden behind this interface. That’s use cases and entities to name just two aspects which are less tangible.

Whereas interface aspects can be identified pretty easily by looking at hardware or APIs, domain aspects are harder to spot. But one thing should be clear: Activities are not the same as data. That’s why use cases should be viewed as aspects of their own separted from the entities they work on.

And to represent those aspects classes seem to be the module level of choice.

How to find such “soft” aspects? Talk to the business roles, understand their domain. Certainly each role will give you ideas for lots of different such aspects. But that does not mean they all belong in just one module. Split them up!

And be open to the possibility that the role won’t be able to lead you to all of them. As your understanding of a domain evolves you will see patterns of change in traits - and will realize there are more and the existing aspects should be cut differently. That’s what Eric Evans calls “refactoring to deeper insight”.

Universal aspects

In addition to the interface and domain aspects I’d like to point out three more aspects which to me are orthogonal to the ones presented so far:

  • Integration
  • Operation
  • Data

I think they are so fundamental, they even come before the other aspects. They are the main shearing layers because they “evolve in different timescales”.[6]

image

Thinking about user interface or persistence or use cases already is pretty specific. It’s about behavioral aspects.

But behavior needs to be coordinated. And behavior needs to be fed with material to work on. To me this distinction is universal. It comes before any of the above mentioned architectural patterns.

Behavior is produced by logic. Logic is encapsulated by modules I call operations. Operations can do some business calculation, query a database, encrypt data, draw a graph or whatever.

Operation is different from data structures. Sorry, OO-folks ;-) Yes, I think for the past 25 years too much data has been entwined with too much logic. We need to start separating data from logic. Data is a very different aspect compared to behavior. Data is about structure, behavior is about process.

And then all the operations need to be “orchestrated”, put into the right order to produce an overall behavior. I call that integration. And it also is a distinct aspect different from data and operations.

Think about a company: There are documents (data) to be processed by clerks (operation) who are overseen by managers (integration). That’s even the case in software development. A product owner produces user stories (data) to be transformed into code (data) by developers (operation). And the whole process is overseen by a Scrum master (integration).

And it’s an anti-pattern if managers (or leaders) try to do the work of the employees they manage. Remember micro-management? ;-)

In code this happens if an integration module does not just “wire-up” other modules to form a process, but also contains logic to add itself to the behavior.

How to translate these universal aspects? Functions and classes are the starting point for me. Focus them on one of these aspects. Let the distinction between these aspects be the fundamental organizing principle for higher level modules/aspects.

Abstracting aspects bottom-up

So much for finding aspects top-down. But there are only so many you can identify this way. But we need to spot more, many more. Because in the end, each function we write should have just a single responsibility. Because a function is a module, the smalles module there is.

That’s where bottom-up aspect hunting starts.

It’s hard to give a definition of an aspect a priori, but it’s comparatively easy to see it as a pattern once there are traits on the table. It’s like the difference between an artist and a spectator. I find a difficult to create a good painting; whereas “good” just means “I like it” ;-) But it’s easy to recognize one when I see it.

So don’t start with the usual object oriented dance around higher level modules like classes - except for the obvious ones. Don’t play the CRC card game.

Instead start from the bottom up. Start with functions. Which means: Start with genuine behavior. All else will follow from there.

Pick some behavior requested by the customer and decompose it. What are the processing steps necessary to transform input into output plus side effects? Then repeat this for each processing step found - until the leafs of this decomposition tree become so small, their implementation is obvious to you.

imageThe processing steps you identify form flows, data flows. Integrating them into a whole is declarative programming, no logic involved. Only the leafs contain logic and only logic (operations). I call this approach “radical object-orientation” or “OOP as if you meant it”. The result of such behavior decomposition are “objects” as Alan Kay thought about them: logic encapsulated in a way so that it only communicates with the outside through messages.[7]

Anyway, what you get are many functions “without a home”. You are at the bottom of abstraction. Now build up from this higher level modules. Take each function as a trait. Which traits seem to be cohesive? Shared state or usage of the same API are signs of cohesion. Likewise focus on different requirements like functionality vs. efficiency. Or within efficiency performance vs. security vs. scalability. Or within functionality calculation vs. transformation or command vs. query. Or of course: integration vs. operation.

Slice and dice the functions (processing steps) in your decomposition hierarchy like you see fit according to observed patterns. First group them into classes. Then group classes into libraries. If you can nest classes in your language, use it to group classes within classes to form ever more coarse grained aspects. The same holds for functions.

But be careful! Within an aspect the traits should be on roughly the same level of abstraction. Abstracting aspect clothes from color and style is fine. But clothes color and collar form seem to be traits on different levels.

This is were experience is needed. Beyond hard and fast rules like suggested above there is a large area where the cohesion between traits and their level of abstraction need to be explored. Try to group them this way, then that way, and feel what makes more sense. Don’t try to avoid “refactoring to deeper insight” later on. Often you can’t be right the first time.

The one reason to change

So where are we after all of this? What about the “one and only one reason to change” which is the hallmark of a module’s responsibility?

You have to learn to spot traits in your code. Like read, write, command, query, calculate, store, data, behavior, encryption, integration, operation, caching, communication, persistence, catalog, billing, logging etc. etc. First see those traits, coarse grained and fine grained, functional as well as non-functional, domain specific vs. cross-cutting.

And then… compare them. Compare them with regard to what causes them to change. And if they have a common cause compare them with regard to how likely it is they change at the same time.

If two traits have a common cause for change and are likely to change at the same time then encapsulate them in a module on an appropriate level. You have defined an aspect and the module’s single responsibility is to implement it.

Learn to do this bottom-up. But don’t set recommendations for common aspects at naught. You find them in patterns and architectural models. Then you can hunt for aspects top-down.


  1. No, don’t even think of metrics like cyclomatic complexity or afferent coupling etc. They do not unambiguously measure evolvability. Also no customer orders evolvability in terms of such metrics. “I want functionality F, with performance P, and evolvability E.” Customers just silently assume the software they order to be easily changeable for an indefinite time. And if that’s not the case, if legacy code problems creep up and slow down development, they are always surprised. As Dag Sjøberg, a Norwegian professor in software engineering, suggests, the only metric to predict evolvability is the number of lines of code. So what customers could say is: For given functional and efficiency requirements keep the LOC at a minimum. But even that I haven’t heard of from customers. They are simply not really aware of the fundamental problem of keeping code bases evolvable.

  2. Ok, maybe with the exeption of functions. Functions where invented to save memory which might have an impact on performance, because more logic can be kept in RAM.

  3. Servers on the other hand, not (µ)Services, serve efficiency requirements like scalability or security. Servers are not modules. They are runtime containers to enable non-functional requirements and I call them hosts to distinguish them from modules. In software architecture they should not be confused with modules. They require independent design consideration.

  4. Hairdo and clothes are pretty obvious aspects, I guess. But if aspects are not hard facts, but abstractions resulting from observations and as such pretty subjective, couldn’t there be an aspect where hair style and clothes style are highly cohesive? Yes, of course. Just think of when hair style and clothes style pretty much change in sync due to a common cause. When is that the case? Say, a child becomes a teenager. Or an adult changes from hippie to conservative. Or a manager becomes a monk. How could this aspect be called when hair style and clothes style (and probably other traits as well) change in unison? How about “worldview”?

  5. If you think, aspects as abstractions are like classes, you’re right. Classes were introduced to be able to bind together what belongs together, to formally express cohesion. But classes are not the only means to define aspects. Aspects can be smaller than what should be encapsulated by a class, and they can be larger. That’s why we should understand modules to form a hierarchy like the one described above.

  6. Which is also true for many aspects, especially the coarse grained. Interface aspects evolve in a different timescale than domain aspects; the UI interface aspect evolves in a different timescale than the persistence aspect. etc.

  7. But I don’t want to delve into this right now. Let it suffice to say: This is not the functional decomposition from the last century. It’s a different kind of design approach, it’s flow design.

In need of more abstraction

The ultimate product of software development is this: CPU executable binary code.

image

Decades ago we used to “write” this more or less directly into memory. But that was very tedious and error prone. Code was hard to reason about, hard to change.

Abstractions in code

So we looked for ways to make coding easier. Enter a higher level of abstraction: Assembler.

image

By representing machine code instructions as text and throwing in macros productivity increased. It was easier to read programs, easier to think them up in the first place, and they were quicker to write with less errors.

According to Jack W. Reeves Assembler source code was a design of the ultimate code which got built by an automatic transformation step.

Soon, though, software was deemed hard to write even with Assembler. So many details needed to be taken care of again and again, why not hide that gain behind some abstractions?

That was when 3GL languages were invented like Fortran or Algol, later C and Pascal etc. But I want to paint this evolution differently. Because from today’s point of view the next level of abstraction on top of Assembler is not a 3GL like Java or C#, but an intermediate language like Java Byte Code (JBC) or the .NET Intermediate Language (IL).[1]

image

Solving the problem of overwhelming details of concrete hardware machines was accomplished by putting a software machine on top of it, a virtual machine (VM) with its own machine code and Assembler language.

Where Assembler provided symbolic instructions and names and macros to abstract from bits and bytes, VMs for example provided easier memory management. Not having to deal with CPU registers or memory management anymore made programming a lot easier.

Now IL Assembler source code was a design for IL byte code, which was source code for machine code Assembler, which was source code for the ultimate machine code. Ok, not really, but in principle.

IL made things simpler - but not simple enough. Programming still left much to be desired in terms of readable source code and productivity. Partly the solution to that were libraries. But also another level of abstraction was needed. Enter 3GLs with all there control structures and syntactic sugar for memory management and sub-program access.

image

That’s where we are today. Source code written in Java or C# is the design for some IL Assembler, which is the design for IL VM byte code, which is the design for machine code Assembler, which is the design for the ultimate machine code. OK, not really, but in principle.[2]

Abstraction beyond code

We like to think of 3GL source code as the design for the executable machine code. As it turned out, though, yesterday’s design, yesterday’s source code became today’s target code.

Yesterday’s abstractions became today’s details. Nobody wants to reason about software on the level of abstraction of any Assembler language. That’s why Flow Charts and Nassi-Shneiderman diagrams were invented.

And what was pseudo-code once, is now a real programming language.

Taking this evolution as a whole into view it begs the question: What’s next?

There is a pattern so far. As many levels of abstractions as have been put onto each other there is one aspect that hasn’t changed. All those languages - Assembler, IL, 3GL - are all about control flow.

Mainstream reasoning about software hasn’t changed. Today as in the 1950s it’s about algorithms. It’s about putting together logic statements to create behavior.

So how can this be extended? What’s our current “pseudo-code” about to be turned into source code of some future IDE?

My impression is: It’s over.

Control flow thinking, the imperative style of programming is at its limit.

There won’t be another level of abstraction in the same vain. I mean language-wise. The number of frameworks to be glued together to form applications will increase. There will be more levels of abstractions.

But to actually design behavior, we will need to switch to another paradigm.

Accessing data has become hugely more productive by the introduction of declarative programming languages like SQL (and modern derivatives like Linq) or Regular Expressions.

So my guess is, we need to go more in that direction. Programming has to become more declarative. We have to stave off imperative details as long as possible.

Functional Programming (FP) seems to be hinting in that direction. Recursion is a declarative solution compared to loops. Also simple data flows as f |> g in F# have declarative power because they leave open whether control flows along with data. f could (in theory) still be active while g already works on some output from f.

Still, though, even with FP there is one question unanswered: How do you think about code?

Is there a way for us to express solutions without encoding them as heaps of texts right away? Is there a way to communicate solutions without and before actually programming them? Can we describe software behavior in a systematic way on a next level of abstraction - and then systematically translate this description into Groovy or Haskell?

Object-orientation (OO) has given us more ways to describe data structures than most developers know. Think of all the relationship types defined in UML.

But software firstly is not about data structures, it’s about functionality, about behavior, about activities. How can that be described, planned, designed above today’s source code, even FP source code?

Because if Assembler, the code design of the 1950s, nowadays is just the output of a compiler translating today’s 3GL source code design… then what kind of design can be translated into today’s source code as a target?

Model-Driven Software Development (MDSD) seems to be trying to answer this question. But despite all efforts it has not been widely adopted. My guess is, that’s because the design of a modelling language is even harder than the design of a decent framework. Not many developers can do that. Also, not many domains lend themselves to this. And it’s not worth the effort in many cases.

But still, MDSD has gotten something right, I guess. Because what I’ve seen of it so far mostly is about declarative languages.

So the question seems to be: What’s a general purpose way to describe software behavior in a declarative manner?

Only by answering this question we’ll be able to enter a next level of abstraction in programming - even if that currently only means to enable more systematic designs before 3GL code and without automatic translation.

We have done that before. That’s how we started with object-orientation or querying data. First there was a model, a way of thinking, the abstraction. Then, later, there was a tool to translate abstract descriptions (designs) into machine code.

The above images all show the same code.[3] The same solution on different levels of abstraction.

However, can you imagine the solution on yet another level of abstraction above the 3GL/C# source code?

That’s what I’m talking about. Programming should not begin with source code. It should begin with thinking. Thinking in terms of models, i.e. even more abstract descriptions of solutions than source code.

As long as we’re lacking a systematic way of designing behavior before 3GL source code - be it OO or FP - we’ll be suffering from limited productivity. Like programmers suffered from limited productivity in the 1950s or 1990s before the invention of Assembler, IL, 3GLs.

And what’s the next level of abstraction?

In my view it’s data flow orientation. We have to say goodbye to control flow and embrace data flows. Control flow will always have its place. But it’s for fleshing out details of behavior. The big picture of software behavior has to be painted in a declarative manner.

Switching from OO languages to FP languages won’t help, though. Both are limited by textual representation. They are great means to encode data flows. But they are cumbersome to think in. Nobody wants to design software in machine code or byte code. Nobody wants to even do it in Assembler. And why stop with 3GLs?

No, think visually. Think in two or three or even more dimensions.

And once we’ve designed a solution in that “space”, we can translate it into lesser textual abstractions - which then will look differently.

This solution

image

surely wasn’t translated from a design on a higher level of abstraction. How the problem “wrap long lines” is approached conceptually is not readily understandable. Even if there were automatic tests to be seen they would not explain the solution. Tests just check for certain behavior.

So, as an exercise, can you imagine a solution to the problem “Word Wrap Kata” on a higher level of abstraction? Can you depict how the expected behavior could be produced? In a declarative manner?

That’s what I mean. To that level of discussion about software we have to collectively rise.

 

PS: Ok, even though I did not want to elaborate on how I think designing with data flows can work – you find more information for example in my blog series on “OOP as if you meant it” –, I guess I should at least give you a glimpse of it.

So this is a flow design for the above word wrapping problem:

 

image

This shows in a declarative manner, how I envision a process for “producing” the desired behavior. The top level/root of the hierarchical flow represents the function in question. The lower level depicts the “production process” to transform a text:

  • First split the text into words,
  • then split words longer than the max line length up into “syllables” (slices).
  • Slices then are put together to form the new lines of the given max length.
  • Finally all those lines are combined into the new text.

This sounds like control flow – but that’s only due to the simplicity of the problem. With slight changes the flow design could be made async, though. Then control would not flow along with the data anymore.

The data flow tells a story of what needs to be done, not how it exactly should happen. Refinement of a flow design stops when each leaf node seems to be easy enough to be written down in imperative source code.

Here’s a translation of the flow design into C# source code:

 

image

You see, the design is retained. The solution idea is clearly visible in code. The purpose of Wrap() is truly single: it just integrates functions into a flow. The solution can be read from top to bottom like the above bullet point list. The code is “visually honest”.

Such a flow design can be easily drawn on a flipchart. With it I can communicate my idea of how to create a certain behavior quickly to my fellow team members. It’s easy to translate into code. And since it does not contain imperative logic, it leads to very clean code, too. Logic is confined to functionally independent leafs in the decomposition tree of the flow design. Read more about this in my blog series “The Incremental Architect’s Napkin”.


  1. I even remember P-Code already used in the 1970s as the target of Pascal compilers.

  2. Of course, this is not what happens with all 3GL languages. Some are interpreted, some are compiled to real machine code. Still, this is the level of abstraction we’ve reached in general.

  3. Well, to be honest, the first image is just some arbitrary binary code. I couldn’t figure out how to get it for the Assembler code in the second image.

The Steep Curve of Feature Prioritization

How to prioritize features for development? Which to do first, which then, which last? That’s a much debated aspect of software development. And most teams I know are not very systematic about it.

That’s a pity, because doing features in the “wrong” order means creating value for the customer slower than possible. Or it even means producing waste.

In his recent book “The Nature of Software Development” Ron Jeffries showed how he thinks prioritization should be done with nice drawings like this:

image

Each rectangle represents a feature with a certain value to be produced with a certain effort (time, money).

image

The higher the “feature box”, the more value is produced. The longer the box, the more it costs to produce the value.

As Ron Jeffries’ drawing clearly shows, there are several ways how to order (prioritize) features. In the end, the same value is produced over the same time. But how fast how much value is added may differ greatly.

Prioritize by Value

His suggestion is: Implement high value features first, postpone low value features as long as possible. This makes for a steeper value growth curve. In the above drawing the top order of features is to be preferred. It grows value faster than the bottom order.

I agree.

But is it really that easy? Just order features according to value and effort? Take this assortment of features for example:

image

Features s..x are of the same value - but require increasing effort. Features d..a are of decreasing value - but require the same effort. The order in which to build thus is in decreasing value and growing effort.

image

I think, this is what Ron Jeffries had in mind:

Look at the difference in the growth of value if we choose the higher value inexpensive features first and defer lower value costly features until later.

But, alas, features usually don’t come in such nicely cut variations. The range of values and efforts is larger. Take the following features for example:

image

Building them in this order leads to this feature value growth curve:

image

Still not bad, isn’t it. A concave curve showing how value is produced quickly right from the start.

Prioritize by Weight

However… We can do better. And that’s what’s missing from Ron Jeffries’ book. Ordering features by value and effort easily leads to suboptimal curvature.

Let me show you what I mean. Here’s an alternative ordering for the same features:

image

Value growth looks steeper, doesn’t it?

How is this? What’s the criteria for ordering the features? It’s obviously not just value, because for example feature c is done before x which provides higher value.

The criteria is called weight. It’s calculated as described by the Weighted Shortest Job First (WSJF) method. It takes into account not only value, but also effort.

weight = value / effort

The higher the weight, the higher the feature should be prioritized.

In a diagram weight can be found in the angle of the diagonal of a feature box. Or to be precise: Since we don’t know the angle, it is the tangent of the angle.

image

The larger the angle, the steeper the diagonal points upwards, the higher the weight, the earlier the feature should be implemented.

What this means for prioritizing the above features shows the following figure where the features are ordered according to their weight:

image

You see: the angle progressively becomes smaller, the inclination of the diagonals decreases. And that means, the value growth curve is steeper, when you implement the features in this order.

Compare the value-focused prioritization drawn with triangles

image

to the WSJF prioritization:

image

The growth of value is steeper and smoother with the WSJF prioritization, there are no slumps in the curve.

Of course, value-focused prioritization and WSJF prioritization result in the same order for features of same effort. So the question is: Can you slice down requirements to be of the same effort - and still “calculate” a meaningful value for each of them?

I’d say, that’s possible - but only pretty late in the game. You already have to be very clear about a lump of requirements to break it down into (roughly) equally sized increments.

That means, mostly you’ll need to prioritize the hard way and do it in WSJF manner.

Finding Value

However, regardless how you prioritize, you need to find the value of your features. How to do that?

In my experience it’s rarely possible to find a monetary value. For large features (whole applications or modules) this might work, but not for smaller features. How many more customers will buy your software if you fix this bug or add the PDF export? How many customers will cancel the subscription to your online service, if you don’t fix the bug or improve usability?

“Cost of delay”, i.e. how much money you’ll loose/not earn as long as a feature is missing, is very hard to determine. I can’t think of any client of mine who would know (on a regular basis).

So what’s the alternative? Choose whatever fits the bill. If you’ve several thousand users you can speculate about how many of them will benefit from a feature. Or even with a few users you can ask yourself, how often they would use that feature on average (each day or each year). Or you can think about how crucial a feature is for their business. Or you can check for dependencies between features; a feature other features depend on might be worth more, because its kind of enabling. Or maybe lacking a feature poses a tangible risk, because the software would loose an important certification.

Find any number of criteria applicable to your software. And assign values to them. You can go with scales from 1 to 5 or the Fibonacci numbers. It’s less about precision as it is about comparability.

Maybe you find just three criteria. That’s better than just a gut feeling. If you range each from 0 to 5 the minimum value for a feature is 0 and the maximum value is 15.

Even with these few criteria talking about feature value becomes more differentiated and objective. Less guessing, more thinking.

Determining Effort

The same is true for determining the effort needed for a feature. We’re talking about estimation here. A sensitive topic for software developers and managers alike.

Calculating the time needed to do a feature with high reliability is near impossible. Ron Jeffries makes that very clear in his book and I can only agree.

So we should not try to predict absolute effort. Fortunately for prioritization it’s sufficient to just determine efforts for comparison. This feature will take x amount of time, that feature will take two times x, and another will take just half x.

Again a simple scale will do. Or go with the Fibonacci numbers again. Yes, it’s almost like with Story Points in Scrum. But don’t fall into the prediction trap! Don’t try to forecast how many features you’ll be able to deliver in a certain amount of time.

As soon as you start forecasting there will be people who take this for (future) reality and depend on the forecast to become true. That reduces your flexibility, that creates pressure. So by all means: Don’t forecast! Take effort figures as abstract numbers just for comparison. Assigning a 2 today will not mean the same when assigned to another feature next week.

Try to do it like this: When determining the effort for a number of features start by finding the smallest one. Assign to it 1 as the effort. Then determine the relative factors for the other ones, e.g. another feature takes twice as long (effort becomes 2), yet another feature takes much, much longer (effort becomes 10) etc.

As should be obvious: The smallest feature in today’s prioritization round can be much smaller or much larger than the smallest feature in the next round. So a 1 cannot be converted to a certain number of hours or days.

Instead of promising a result (“We will deliver features a, b, c in one week’s time.”), you should just promise a behavior (“We will deliver features a, b, c in order of their priority - which might even change over time.”).

Remember: It’s always good to promise like a pro! ;-)

Bottom line: When building software go for incremental value delivery. But value alone is not enough to prioritize. You need to take effort into account. You achieve the steepest growth in value when you prioritize based on feature weight, which essentially means you calculate a speed: “value delivery speed” equals feature value by implementation time. Bet on race horse features first, leave the nags for last.

Feedback-Centric Development - The One Hacker Way

Erik Meijer got something right in his talk "One Hacker Way". There's a lot of bashing and ranting... but at the core there also is a precious diamond to be found. It's his admonition to be driven by feedback.

As software developer we should focus on production code - and let ourselves be guided by feedback.

How true! How simple! But contrary to the audience's belief it's no easy feat. He got much applause when he suggested, attendees who had not committed code recently should leave. People liked him extolling the virtues of "hacking", of focusing on code - instead of on fuzzy stuff like a process or even talking or thinking. No, it's the code, stupid!

Unfortunately they did not get the implications of this, I guess. And Erik Meijer did not tell them what that really, really means. So I'll try to describe how I see what truly and honestly focusing on code and feedback means.

Purpose

I'm sorry, but before I get to code, we need to lay a foundation. We need to be very clear about why we should produce code in the first place.

Code is a tool for our customers. Customers want to use software to achieve something, to reach a goal.

In order to be helpful, code needs to fulfill certain requirements. I see three basic requirements:

  • Software needs to be functional, e.g. a calculator has to provide addition and multiplication.
  • Software needs to be efficient, e.g. a calculator has to add and multiply much fast.
  • Software needs to be evolvable, e.g. a calculator needs to be adaptable to changing functional and efficiency requirements, maybe it has to also provide sine operation or has to become even faster.

Functional and efficiency requirements define the behavior of software which is produced by logic (for me that's transformational statements, control-flow-statements, and hardware access). Evolvability requires a certain structure which is spanned by modules of several sizes (for me that's function, class, library, component, micro-service).

What Erik Meijer means, when he favors hacking over some obscure agile way of development is, that software developers should produce code in order to create appropriate behavior and structure.

And what he means, when he says we should look for feedback is, that we should check whether the code written already delivers on the behavioral and structural requirements.

Being Feedback-Centric

Now for the fun part: If Erik Meijer is serious about feedback, he needs to emphasize that it has to be sought frequently. In fact the central and recurring question everything is revolving on is:

How can I get feedback as quickly as possible on my code?

That's what I call feedback-centric. Yes, we should focus on code. But code without feedback has no value. So we should seek feedback "at all costs". As soon as possible. Frequently. From the most relevant source.

Software development thus becomes a high frequency iterative activity:

  1. Code
  2. Check (gather feedback)
  3. Back to 1. for fixing any deficit or add more behavior/structure

Feedback-centric development thus is code-first development. I like. Don't you? As Erik Meijer said: Forget about test-first programming or even TDD. It's the production code which rules!

Tiny Steps - The No. 1 Implication

If you really, really buy this - it's about code and about feedback -, then you also have to buy the implication: Coding has to progress in tiny steps.

Because only tiny steps can get you frequent feedback. If you hack away for an hour or a day without feedback, then you're coding pretty much in the dark. Truely frequent feedback is not more than a couple of minutes away.

When you look at some requirements you have to ask yourself: What can I do to get feedback in the shortest possible amount of time? Can I feedback in 10 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes?

Not Only Code - The No. 2 Implication

The ultimate feedback of course is on code. So if you can feedback on some code in 1 minute go for it.
 
At least initially, though, it's even faster to get feedback without any code. Producing code and getting some stakeholder to check it for conformance to requirements often takes longer than simply asking a question.

Code should not be a question, but a statement of some understanding - even if that turns out to be wrong.

So as long as you've questions or are not very sure to understand what the requirements are... do not start hacking. Rather ask questions, e.g. by talking or by presenting some test cases you made up.

Incremental Steps - The No. 3 Implication

Being driven by code and feedback also means, you can't just programming any code. The code you want to write is, well, code you can get feedback on. That means it needs to be code some stakeholder can relate to.

Feedback-centric development thus means producing code incrementally. Code needs to make a difference, needs to produce some possibly very small additional value. And if that's indeed the case only a stakeholder can tell you.

Automatic Tests - The No. 4 Implication

Once you've identified a tiny increment you can start coding. That's just fine. No need to write a test first. What a relieve, isn't it? ;-)

Then, after maybe 3 minutes of writing production code, you run the code to give yourself a first round of feedback. Since you've asked a lot of questions you're somewhat of an authority on the requirements - but of course by no means ultimately decisive. The right to accept only lies with stakeholders.

But how do you run the code and check it for deficiencies? You can do that manually. That's just fine. But how frequent can you then check?

Not checking the behavior for correctness with automatic tests is a violation of the core principle of feedback-centric development. See, it's not just code, but also fastest feedback possible. You have to balance code production and feedback generation.

That means, you need to write automatic tests. Do it after you wrote your production code. That's fine. Since you only added a tiny increment there is not much to test. At least do it for every error you encounter. Reproduce the error with an automated test, then fix it. Rerun the test to get feedback if your fix actually fixed it.

Automatic tests have two feedback purposes:

  • whether the code you just wrote delivers on the required behavioral increment
  • whether other code still delivers on its behavioral requirements (regression tests)

TDD might seem to provide no benefit. But it should be clear now, that test-after, i.e. for feedback generation after hacking is a must. It's a sine qua non if you're serious about the One Hacker Way.

Testable Structure - No. 5 Implication

Now that automatic testing finally is inevitable even for the most eager hacker ;-) it should be obvious that not just any code structure will do. The code must be structured in a way as to be easily testable.

That even means, each increment should be testable in isolation. Otherwise the feedback would not be precise - which would be a violation of a fundamental principle we started out with.

What this leads to is... refactoring. Finally! Because in TDD refactoring is clearly optional. Yeah, it's a prescribed step after the test went green - but look at the TDD examples out there. They are a testament to how easy it is to jump this step.

No, TDD does not (!) exert any force to make a developer refactor her code. Everyone rather writes the next red test.

But if you're serious about the One Hacker Way, i.e. feedback-centric development, then you have to provide yourself with quick and precise feedback. And that (!) requires you to structure the code in a way to make it possible.

  1. Code some increment
  2. Write a test to get feedback on just that increment; if that's not possible, refactor enough to get it working

Feedback-centric development makes you the first consumer of your code. Eat your own structural dog food and see if it's palatable, i.e. if you can easily test the logic hanging in that structure.

Structural Review - No. 6 Implication

Manual or even automatic tests just provide feedback on behavior. But as stated above it's not just behavioral requirements the production needs to deliver on. Customers want us to write code in a sustainable way. Nobody knows what kind of changes come around in the next weeks, months, years. So our production code needs to be prepared; it needs to be evolvable.

Evolvability is a quality of the structure of the code. Traditionally it's produced by some kind of modularization. Recently some more principles have been added to reach this goal under the name of Clean Code.

However you call it one thing is for sure: evolvability is hard to measure. Automatic tests and the customer/user can comparatively easy give feedback on functionality and efficiency. But whether evolvability is high enough... they can't tell. Especially because evolvability cannot be stated in a requirements document.

Customers simply assume software to be infinitely malleable and live forever. Mostly, at least to my experience.

That means, tools measuring certain structural metrics cannot undoubtedly truth about the structural quality of software. At best the might hint at certain spots where it seems evolvability is lower than desired.

The ultimate feedback on evolvability comes only from... developers. If developers have a hard time to change a codebase, then it's hard to evolve. That simple.

How then can feedback from developers as authorities on evolvability be gathered frequently?

Firstly, the feedback is generated implicitly by adding the next increment. If the developer trying that find it difficult, he just has generated feedback - and can act on it. Refactoring is fixing an evolvability deficiency when it arises.

Unlike with TDD where there is no feedback on structure generated, and refactoring is recommend in a broad brush manner, in feedback-centric development refactoring always has a clear purpose. It's done when necessary to enable the next increment.

Evolvability is too important to leave it to a single developer, though. Sensitivity to structural quality is very unevenly distributed among developers for several reasons. That's why is helpful to get feedback from more than one developer as soon as possible.

Enter pair programming. During pair programming it's possible to focus on behavior and structure at the same time. Four eyes see more than two. So if you haven't been convinced of pair programming so far, but like the idea of The One Hacker Way... now is the time to start pair programming. It's a valuable technique to get more frequent feedback on code structure.

Equally valuable is of course the age old technique of doing code reviews. I don't think they should be replaced by pair programming. Code reviews go beyond the four eyes of the developers who wrote the code. More eyeballs simply can spot more structural flaws. Also a group can check if the structure matches a common understanding of how code should be modularized.

Even with pair programming and code reviews I feel there is something missing, though. They generate feedback on structure with different frequencies and from different perspectives. But the feedback of both is, hm, somewhat artificial.

Improving the structure to enable an automatic test carries a certain urgency. Refactoring is really needed to be able to continue according to the principle of frequent feedback. Pair programming and code reviews don't "force"  structural improvement in such a way.

That's why I suggest another technique I call code rotation (or maybe story rotation). Core rotation means, some requirement should not be fully implemented by a single developer or even pair. If coding an increment takes a day, for example every 90 minutes the eyeballs looking at it should be completely exchanged. Maybe developer A and B start, then C and D continue etc. Yes, A and B should be replaced by a fresh pair. There is a quick handover - and then the new pair has to get along alone with the codebase.

And that's the point: Only if the developer(s) working on a requirement change completely will the be honest feedback about the structure. If even one dev of the first pair remains for a second tour on the code the feedback is "contaminated". We're so prone to lie to ourselves when it comes to our own code... This can only be avoided by letting fresh eyeballs look at it.

Sometimes I employ this technique in trainings. I let developers start to work on an exercise - and then after a while they hand their code over to their neighbour on the right. You can imagine that this is no delight for anyone ;-) But why? It's this dissonance that needs to be removed from coding. It stems from non-obivous code structures.

Bottom line: Evolvability is of great importance. Unfortunately it's hard to measure. So we need to actually look at code and work with it to get a feeling for its quality. Therefore we need to establish a hierarchy of feedback cycles:

  1. Make every increment testable - refactor as needed. Frequency: continuously
  2. Make code understandable to your pair programming partner - refactor as needed. Frequency: minutes
  3. Make code understandable to your successor - refactor as needed. Frequency: hours
  4. Make code understandable for the whole team - refactor as needed. Frequency: day(s)

Software Design - No. 7 Implication

Switching pairs during development of a feature is a tough call. Sure you want to avoid it. But why? Too much context switch? Takes too long to find your way around the code of other devs to be able to continue their work?

Yeah, right. That's all difficult. But you can choose: experience that now - or sometime in the future. And it should be obvious that it becomes harder the longer it takes until somebody else looks at your code.

In order to make code rotation as smooth as possible another technique is needed. Frequent feedback gets another prerequisite: design.

Yes, I believe the reason for explicit design now should becomes apparent. Explicit design done by a group of devs or even the whole team helps to understand the code. Also it decreases the need for refactorings later.

Some modularization cannot be avoided to be done ad hoc during hacking. But quite some modularization can be done before even starting to code. It's a part of developing a solution. It's the "thinking before coding". And it has value because it makes it easier for developers to switch working on the codebase.

So forget about design "because that's how you do software development". Also forget about not doing design "because that's how real programmings code."

Explicit design is a means to an end. It's purpose is to develop a common understanding of easily evolvable coarse grained structures - in order to increase the frequency of feedback. Because you don't want to wait years to realize you're sitting on a monolith, if the next developer can tell you in a couple of hours he has a hard time extending what you left behind.

Continuous Deployment - No. 8 Implication

Ultimate feedback only comes from those who require your code to do their job. That means it must be very, very easy to give your code to them. The closed the code to the final usage environment the better.

That's why continuous deployment is so important for any software project. We need it to be a no brainer as much as possible to deploy code so we can ask just about anybody for feedback at any time.

Think about A/B deployment, think about deploying each increment separately, think about deploying only to a subset of customers... The more freedom and options you have, the better for gathering feedback.

In Closing

At first I did not like Erik Meijer's talk much. But once I saw through his polemic fireworks I realized how much truth can be found in what he said. Never mind his suggestion to treat developers as top athletes. Never mind him calling Jeff Sutherland a satan.

Let's stick to the title, the core of his message: We need to focus on code - because only that's delivering value. And we need integrate feedback into our work much more seriously - because only then we know if we're actually heading in the right direction with our code.

Forget about hype, buzzwords, and any elaborate belief system like "Agile" or "Scrum" etc. Yes, like the Buddhists are saying: "If you meet the Buddha on a road, kill him." We need to kill our Buddhas, the gurus, the dogmas. Let's do away with cargo cults.

Instead focus on the essential: production code. And get as much feedback as possible. Truely become a closed system on many levels of your daily practice and your organization.

PS: If you happen to recognize one of your favorite “agile practices” in my above description, congratulations. Of course there is value in some of them. We don’t need to throw the baby out with the bath water. My point, though, is to justify such practices starting just from production code and the need for feedback. Nothing more, nothing less. No buzzwords, no argumentum ad verecundiam.

Software development must deliver on budget - always

Yes, I mean it: we always need to meet the budget (be that time, money or whatever resource).1

This most likely is not your software development reality. So how come I´m demanding something so seemingly unrealistic, even preposterous?

Why?

The reason for the obligation to deliver on budget is simple: trust.

Software development is a social endeavor. It takes not only two to tango, but also at least two to develop and deliver software: a customer and a software developer.

To accomplish something in collaboration with other people requires trust. Trust is the foundation because you cannot do everything yourself. You need to let go of something and trust a collaboration partner. That´s the very reason for collaboration in the first place. If you were able to do something yourself, why get someone else on board?

So if there is a need for cooperation, then there is a need for trust. Even more so if the relationship between the cooperating parties is highly asymmetric: If you could do something yourself but delegate it to somebody else, you can at least check their work for quality. Less trust is needed.

But if you don´t have a clue how to accomplish something yourself, you can´t help but delegate execution and on top of that you´re unable to check every detail of how the result is produced. You´re pretty much limited to checks on the surface. So you need much more trust in this situation.

Software development mostly is a highly asymmetric endeavor. Customers don´t understand much of it, so they need to trust software developers.

How is trust created? Trust grows through reliability and steadiness. It´s delivering on promises - and even going beyond that. If your cooperation partner does what she promised, she´s reliable. If she does that again and again... you learn to trust her.

And if someone presents you with value you had not even expected... well, that´s even better.

Gifts build trust, keeping promises builds trust.

Delivering below budget is like a gift. Delivering on budget is keeping a promise.

That simple.

What´s wrong?

But why is it so common project reality to not deliver on budget? There is something fundamentally wrong with how the industry (or its customers) interprets the term "promise", I guess.

Promises are a form of contract. A contract requires all parties to freely agree on it, which requires all parties to understand the contractual clauses (in the same way) and their implications and ramifications. And a contract requires all parties to be able to live up to it.

Contracts are the foundation of civilization. The Romans recognized that and coined the term: pacta sunt servanda.

So what goes wrong in our industry with promises?

  1. Software development often is not free to enter a contract. Management states "You deliver such scope on such budget!" - and thinks a contract exists. But it does not. If a party got coerced then it´s not actually part of the contract.
  2. Software development often does not fully understand what´s in the contract, what it implies. Some parts of the contract are clear (mostly the budget), some are not clear, fuzzy, or outright not understandable (mostly the requirements). How can software development then honor the contract?
  3. Software development regularly overestimates its capabilities to deliver on what it perceives to be in the contract. It lacks experience, it neglects dependencies etc. Even though software development freely enters a contract or even defines the budget, it is unable to deliver on it.

No small wonder, so many contracts - large and small - are not honored. Promises are broken. Trust is not build or erodes. Overall quality drops to save at least a few contractual goals. Conflicts and pressure ensue.

How?

The only remedy to this recurring downward spiral, to this pattern is: Always deliver on budget. Always live up to what you promised.

But how?

I think, it´s very simple - which does mean it´s easy ;-)

Only make promises you can keep.

Ask yourself: Am I willing to bet 1000€ I will be able to keep this promise? If not, well, then you´re not in a position to make a promise. You doubt your own reliability.

This simple and obvious rule of course has a couple of implications:

  1. Spot and withstand any coercion. Do not enter any contract you have the faintest doubt whether you can deliver on - just because someone says so.
  2. Be humble with regard to your capabilities. Don´t overestimate what you know or are able to do.
  3. Be realistic in terms of control over your environment and your time. I strongly believe you cannot control more than maybe 1 or 2 days of your time. So you can´t promise anything beyond that.

So what can you promise? Stuff you´ve done already many times; that´s then called reproduction. You can only make promises on what you have considerable experience with and thus can deliver in "reproductive mode".

If you have experience with cooking Chicken Vindaloo you make a promise to deliver in maybe 60 minutes. But how far goes your experience in software development? Even if you´ve been doing it for 25 years your ability to reproduce is limited. This is not just because of ever changing technology, but because of ever changing customer requirements. They are hardly ever the same. As is your software.

Software is like a river: you can´t enter a river twice. Which means, it´s not the same river the next time you step into it. Something has changed, so have you.

Software development is hardly ever in reproduction mode. Mostly it´s a creative undertaking. Stuff needs to be analyzed, researched. Stuff needs to be engineered, invented.

So what can you promise to deliver on budget?

You can only promise your diligence. You can promise to work hard, to focus. But you cannot promise the implementation of a certain scope (functionality, quality) in a particular timeframe (budget). At least not beyond very small increments.

So here´s the bottom line:

  • Promise concrete results only in matters of reproduction.2
  • Since the circumstances for reproduction are rare, be conservative. Even if you are able to reproduce a result, it becomes difficult beyond 1 or 2 days - because you hardly have control over your time.
  • Without the ability to reproduce results promise only one thing: focus. Focus on a contractual topic regularly until the contract is fulfilled. Be persistent, be relentless - but don´t expect to know in advance when you´ll be done. Which means, be honest about this; don´t let yourself be lured into promises you can´t keep.

We can´t avoid to make promises. Promises are the foundation of collaboration. So we must stick to promises we actually can keep. Be extra careful what you promise. With regard to coding the time budget you agree on should be very, very short. And once you´re beyond reproduction mode, only promise diligence.

Who wants to work with unreliable people? Nobody. So we need to strive to become more reliable, more trustworthy. Working with software development should become a pleasure.


  1. An overrun of maybe 5% might be ok to classify a result as “within budget”.

  2. Yeah, I know what you´re thinking right know ;-) But there is more than that kind of reproduction…

How Agility leads to functional design and even TDD

What is it that the customer wants when she orders a software? Behavior. I define behavior as the relationship between input, output, and side effects.

It´s like with the Turing Test. When can we consider a machine intelligent? As soon as we cannot tell from a dialog whether the "hidden participant" is a human or not. The Turing Test is about behavior.

Requirements are met if some input leads to desired output and expected side effects. This includes performance, security, usability and other aspects. Behavior thus has functional as well as non-functional traits.

Now the question is: How is behavior produced?

It´s all about logic, program logic. That is operators, control structures and hardware access. Only such programming language statements are relevant to producing behavior by working on data.

Nothing has changed since Niclaus Wirth wrote "Algorithms and Data Structures" back in the 1970s. Nothing even has changed since the days of assembler programming.

Forget about Object Orientation. Forget about Functional Programming. At least for a moment. That´s all just tools, not givens.

The main question in programming is, how to move efficiently and effectively from requirements to logic? You can imagine requirements and logic separated by a huge gap, a chasm even.

image

On top there is the whole of all requirements for a software system. At the bottom there is all the logic that´s needed to show the required behavior. That´s just all operator statements, control statements, and hardware access statements.

Except for trivial requirements we cannot jump over the chasm. For the Fizz Buzz code kata the whole logic might appear immediately before your minds eye. But probably not even for the Bowling Game code kata, and sure not for solving Sudoku puzzles.

That means we need help to cross the chasm.

To me this help comes as a three phase process.

1. Agile analysis

The first phase is about thinking. We need to analyze the requirements. But not just in any way. We need to keep the customer´s world and the developer´s world close together.

Analysis to me means not only to understand what the customer wants, but also to slice it up into ever more fine increments.

Increments are parts/aspects of the overall behavior. The customer can give feedback on them.

User Stories and Use Cases are examples of such increments - but unfortunately they lack connection to the developer´s code reality. What´s the equivalent of a User Story in code?

That´s why I prefer (and suggest) to find more tangible increments during requirements analysis. I call them Application, Dialog, Interaction, and Feature. (There are even two more, but these are the most important ones.)

image

Analysis considers the problem. It´s tries to understand it - also by de-constructing into smaller problems. Analysis is a research task.

I call this kind of analysis agile, because it produces increments. It´s not about technical artifacts, but just aspects the customer can relate to.

To ask whether the requirements should be met by just one Application or more leads to smaller separate problems - and at the same to time artifacts tangible for the developer. An Application can be thought of as a project on any development platform/IDE. It´s represented by an executable file at runtime, an icon on a desktop or a URL to open in a browser.

Requirements can be fulfilled by delivering one Application after another.

The same is true for Dialogs. Each Application consists of a number of Dialogs through which users converse with the logic. Each Dialog delivered provides some value to the customer and can be given feedback on.

For the developer a Dialog is very tangible, too. It´s usually encoded as a class (or module). For GUIs that´s readily done by the IDE.

Dialogs in turn consist of a number of interactions. Buttons can be pressed, menu items clicked etc. There are many technical events happening on a user interface - but some are special. They trigger behavior. In GUI dialogs we write event handlers for that. I call them Entry Points into an application. They are like the Program.Main() functions of Java/C# programs.

Interactions represent "single behaviors": Some input is taken from the user interface, output is produced to be displayed on the user interface, and possibly other side effects happen, e.g. data gets changed in a database.

Interactions are clearly relevant to the customer. They have specific, tangible triggers. Their behavior can be exactly defined (At least it should be. The customer is responsible to provide approval criteria in terms of input/output/side effect relationships.)

But at the same time Interactions are tangible for developers. Their equivalent in code is always a function.

How much better is that than being confronted with a User Story?

User Stories and Use Cases are nice and well - but they should be mapped onto Applications, Dialogs, Interactions before moving to the next phase. Nothing is lost for the customer by that - but much is won for the developer.

And even further the analysis should go! Interactions are cross-cuts through the software. Much can happen while input is moved from the user interface through the bowls of the software to transform it into some output - again presented by the user interface. Such transformation certainly has many aspects to it. These aspects should be considered during analysis. I call them Features.

A Feature is an aspect of an Interaction in a Dialog of an Application. It will be represented in code by at least one function of its own.

Features thus are tangible for the developer - but at the same time are relevant to the customer. Take a user registration Dialog for example. Such a dialog will at least have one Interaction: register user, e.g. when hitting the OK-button.

Taking this Interaction only as a whole and trying to figure out what logic is needed to me seems hard. Better to refine it, better to slice Features off it, e.g.

  • Create new user
  • Check if user name already exists
  • Check if user name is well-formed
  • Check if password is well-formed
  • Check if password repetition equals password
  • Show error message if check fails
  • Encrypt password before storing it

Features are the most fine grained requirements in agile analysis: they are increments and can be directly mapped to code. Software can be delivered Feature by Feature. Logic can be developed Feature by Feature.1

Agile analysis can and should of course be done together with the customer. It does not need to be comprehensive. Avoid big analysis up-front. Some top-down breadth-first then depth-first analysis is sufficient until you have enough Interactions and Features on your plate to let the customer do some prioritization.

Then enter the next phase...

2. Functional design

Knowing which Applications, Dialogs, Interactions, Features there are does not close the requirements-logic-gap. Agile analysis makes it smaller, but it´s still too wide to jump across.

Interactions exist side-by-side. Their connection is through data only. But Features are connected in all sorts of ways within Interactions. Their relationship is causal. The activity of one Feature leads to activity of other Features.

Features form production processes: they take input and data from other resources and transform it into output and data in other resources.

image

The appropriate way of thinking about how Features make up Interactions thus is data flows. Yes, data flows and not control flows. Logic is about control flow; that´s why it contains control structures.

To find the right data flows to deliver the required behavior for each Interaction I call behavioral design or functional design.

"Behavioral design" emphasizes the purpose, the results of what´s happening by data flowing.

"Functional design" on the other hand emphasizes the technical side of data flows, their building blocks, which are functions.

Designing data flows is an engineering task. It takes the results of the analysis phase and tries to solve the "behavioral problem" of an Interaction by combining Features already found with more Features, which only become visible/necessary when looking under the hood. Functional design considers technologies and paradigms to define appropriate data flows.

Of course, such flow designs are "just bubbles" at first. But that´s not a drawback, it´s a feature. "Bubbles" can easily be revised, created, destroyed. "Bubbles" can be visualized, can be talked about among team members with just pen and paper as tools (enter: The Architect´s Napkin ;-).

Behavioral design means solving problems on a conceptual level using a simple DSL: data flows. The syntax and semantics are easy to learn. They provide a framework for phrasing solutions using domain specific vocabulary.

Functional design closes the requirements-logic-gap:

image

The vocabulary in the data flows can then straightforwardly be translated into functions.

Please note: Functions are not logic! Functions are containers for logic. Functional design thus results in a list of containers which then have to be filled with all the logic details that´s necessary to actually deliver the desired behavior.

Data flows on the other hand lack many details. They are declarative by purpose. They describe behavior in a comparatively coarse grained manner. They are abstractions to help find logic containers and to be able to reason about it in a simpler way.

Without data flows it´s hard to understand logic. You get bogged down in an infinite sea of details at no time.

To avoid that during design as well as bug fixing and enhancing software use data flows. Done right they provide a smooth transition from analysis to coding - and even back. Because done right data flows are not just a matter of pen and paper but are clearly visible in code. If you look at data flow code you can "reverse engineer" the data flow design from it.

But still... data flows won´t crash as long as they´re just designs.

3. Test-first coding - finally

The third phase is about coding; finally we´re talking algorithms. It´s imperative programming as you´re used to. But you start with something in your hands: a list of functions that have to be implemented. This means you can really focus on crafting code. It´s small pieces of logic at a time. (To be honest: Sometimes that´s even the most boring part ;-) Analysis and design have narrowed down the scope so much that you can do the final step from requirements to logic. It´s not a “leap of faith” anymore; it´s pretty straightforward craftsmanship.

To manifest data flow designs use test-first coding. I´m not calling it TDD, because there is less or even no refactoring after the red-green steps.2

Data flows are easy to test. The strategy is obvious:

  • Leafs of hierarchical data flows are not functionally dependent on any code you write. I call them Operations; they contain pure logic. That makes them easy to test. No dependency injection needed. That´s pure unit testing.
  • Nodes within hierarchical data flows are not functionally dependent either. Although they depend on other nodes/leafs these dependencies are not functional in nature. That´s because the nodes do not contain any logic at all. Their sole purpose is Integration. Dependency injection might be needed - but it´s pure integration testing, that means tests do not check behavior but "wiring". Integration tests answer the question: Have all parts been wired-up correctly? And since Integration data flow nodes do not contain logic they are small. Often they don´t need to be tested automatically; a review is sufficient.
  • The root node of a data flow hierarchy - often the Entry Point of an Interaction - needs to be checked with acceptance tests. This way it´s ensured the data flow actually produces the desired overall behavior as a whole.

image

Data flow design ensures, all functions are small. Logic is "compartmentalized" in a way to make it easy to understand and test.

Ideally each Operation is as simple as an average code kata. Functional design provides the developer with a function signature and test cases. That´s a very concrete base for driving the work of a craftsman to hammer out the algorithm, the logic.

What about classes?

At the beginning I asked you to forget about Object Orientation and other programming paradigms. Can you now see why?

Software development in general is not about any of it. It´s about delivering logic. And in order to be able to do that it´s about analyzing and designing the solution as a preparation to code. And what to code in the first place is behavior, that means logic contained in functions. That hasn´t changed in all the decades since the invention of subroutines.

Not to start software development by focusing on functions thus leads you astray. Focusing on classes (objects) first is, well, counter-productive.

Classes are containers, containers for functions. Without knowing which functions are needed to produce behavior it´s a waste to look for classes.3

Functional Programming on the other hand puts functions first. That´s good - as long as that means data flows can more easily be translated into code. But don´t get mired in fancy languages features. Talking about Monads or recursion or statelessness can deflect your mind from more important things like delivering value to the customer.

Functional design is not in contradiction with Object Orientation or Functional Programming. Take it more as a framework to use use these tools in. Tools need rules; we shouldn´t do with them all that´s possible but what´s healthy and beneficial in the long run.

That´s why I´m not a fan of pure/traditional Object Orientation or Functional Programming. Such dogma does not lead to frequent delivery of and feedback on behavior. The trend towards hybrid languages is more to my taste. C# (and even Java) becoming more Functional, and F# or Scala being Functional but also supporting Object Orientation seems to the way to go.

Hybrid languages make it easier to translate data flow designs into code with all their aspects, which includes local and shared state.

Summary

Considering all the details of an implementation of required behavior is a daunting task. That´s why we need a systematic way to approach it starting from requirements and leading to logic.

To me that´s a three phase process starting with an agile analysis resulting in fine grained increments meaningful to customers and developers alike. Then moving on to designing a solution from the increment "powder" in the form of data flows, since they allow us to reason about it on a pretty high level of abstraction. And finally coding individual functional units of those data flows in a test-first manner to get high test coverage for the myriad of details.

I´ve been working like this for the past couple of years. It has made my life as a developer and trainer and consultant much, much easier. Why don´t you try it, too? If you´ve any questions on how to start, feel free to write me an email.


  1. If your User Stories are already like what I call Features, that´s great. If not, but you like to stick with the User Story concept try to write them after you´ve uncovered Interactions and Features by agile analysis.

  2. This is of course not completely true. Not all design can be done up-front for an Interaction or even Feature. There are almost always aspects which cannot be foreseen. So you stumble across them during coding. That´s perfectly fine and does not cause harm. It just leads to an ad hoc extension of design. Because what to do during refactoring is crystal clear: morph the logic just implemented into data flows.

  3. Classes also contain data. Data structures can be build from them. As long as you use them for that purpose, go ahead. But don´t try to fit logic on them at the same time.

The Incremental Architect´s Napkin - #7 - Nest flows to scale functional design

You can design the functionality of any Entry Point using just 1D and 2D data flows. Each processing step in such flows contains logic1 to accomplish a smaller or larger part of the overall process.

To benefit most from Flow Design, the size of each such step should be small, though.

Now think of this scenario: You have a program with some 100,000 lines of code (LOC). It can be triggered through 25 Entry Points. If each started a flow of maybe 5 processing steps that would mean, functional units would contain around 800 LOC on average. In reality some probably would be just 50 LOC or 100 LOC - which would require others to contain 1,500 LOC or even more.

Yes, I mean it: Think of the whole functionality of your software being expressed as flows and implemented in functional units conforming to the Principle of Mutual Oblivion (PoMO). There´s no limit to that - even if you can´t imagine that right now yet ;-)

What should be limited, however, is the length of the implementations of the functional units. 1,500 LOC, 800 LOC, even 400 LOC is too much to easily understand. Logic of more than maybe 50 LOC or a screenful of code is hard to comprehend. Sometimes even fewer LOC are difficult to grog.

Remember the #1 rule of coding: Keep your functions small. Period. (Ok, I made up this rule just now ;-) Still I find it very reasonable.)

The #1 rule of Flow Design then could be: Don´t limit the number of processing steps. Use as many as are required to keep the implementation in line with the #1 rule of coding.2

Flow processing steps turning into functions of some 50 LOC would be great. For 100,000 LOC in the above scenario that would mean 2000 functional units spread across 25 Entry Point flows, though. With each flow consisting of 80 processing steps. On average.

That sounds unwieldy, too, doesn´t it? Even if a flow is a visual representation of functionality it´s probably hard to understand beyond maybe 10 processing steps.

The solution to this dilemma - keep function size low and at the same time keep flow length short - lies in nesting. You should be able to define flows consisting of flows. And you are.

I call such flows three dimensional (3D), since they add another direction in which to extend them. 1D flows extend sequentially, "from left to right". 2D flows extend in parallel by branching into multiple paths. 3D flows extend "vertically".

image

In 3D flows a 1D/2D flow is contained in a higher level processing step. These steps integrate lower level functional units into a whole which they represent. In the previous figure the top level functional unit w integrates s and t. One could say s and t form the w process.

s in turn integrates a, d, and f on the bottom level. And t wires-up b, c, and e to form a flow.

a through f are non-integrating functional units at the bottom level of this flow hierarchy.

Showing such nesting relationships by actually nesting notational elements within each other does not scale.

image

This might be the most authentic depiction of nested flows, but it´s hard to draw for more than three levels and a couple of functional units per level.

A better choice is to draw nested flows as a "tree" of functional units:

image

In this figure you see all levels of the process as well as how each integration wires-up another nested flow. Take the triangles as a rough depiction of the pinch gesture on your smartphone which you use to zoom in on a map for example. It´s the same here: each level down the diagram becomes more detailed.

Most of the time, though, you don´t need to draw deeply nested 3D flows. Usually you start with a top level flow on a napkin or flip chart and then drill down one level. If deeper nesting is needed, you take a new napkin or flip chart and continue there.

Here´s an example from a recent workshop. Never mind the German labels on the processing steps:

image

It´s a functional design on three levels also including the class design. But that´s a topic for another time.

What I´d like you to note here is the sketchy character of the design. It´s done quickly without much ado about layout and orderliness. It´s a "living document", a work in progress during a design session of a team. It´s not a post-implementation depiction (documentation), but a pre-implementation sketch. As that it´s not supposed to have much meaning by itself outside the group of people who came up with the Flow Design.

But it can be taken to explain the design to another person. In that the diagram would be taken as a map to point to and follow along with a finger while explaining what´s happening in each processing step on each level.

And of course it´s a memory aid. Not only talking about a (functional) design but actually keeping visually track of it helps to remember the overall software structure. A picture is worth a thousand words.

Back to LOC counting: With nested flows 80 functional units per Entry Point should not sound unwieldy anymore. Let´s put 5 functional units into a sub-flow for integration by its own functional unit on a higher level. That would lead to 16 such integrating processing steps. They would need another 3 functional units for integration on yet another higher level. So what we end up with is 1 + 3 + 16 + 80 = 100 functional units in total for some 4,000 LOC of logic code. That does not sound bad, I´d say. Admittedly it´s an overhead of 25% on functions - but it´s only maybe around 5% more LOC within functions. As you´ll see, integration code is simple. A small price to pay for the benefit of small functions throughout the code base.

Integration vs operation

You might think, nested flows are nothing more than functional decompositions of the past. Functions calling functions calling functions... But it´s not.

Yes, it´s "functions all the way down". Those functions are not created equal, though. They fundamentally differ in what their responsibilities are:

  • Integrating functional units just do that: they integrate. They do not contain any logic.
  • Non-integrating functional units just contain logic. They never integrate any other functional units. That´s Operations.

I call this the Integration Operation Segregation Principle (IOSP). It´s the Single Level of Abstraction (SLA) principle taken to the extreme. Here´s a flow hierarchy reduced to its dependencies:

image

There is any number of integration levels, but only one level of Operations. Operations are the leafs of the dependency tree. Only they contain logic. All nodes above them do not contain logic.

That´s what makes decomposition in Flow Design so different from earlier functional decomposition. That plus Flow Design being about data flow instead about control flow.

Or let me say it more bluntly: I strongly believe that "dirty code" is a result of not containing logic in a systematic manner like this. Instead in your code base logic is smeared all over the de facto existing functional hierarchies across all sorts of classes.

This subtly but fundamentally violates the SRP. It entangles the responsibility of whatever the logic is supposed to do (behavior) with the responsibility to integrate functional units into a whole (structure). "Pieces of" logic should not be functionally dependent on other "pieces of" logic. That´s what the PoMO is about. That´s what Object Orientation originally was about: messaging.

To fullfil functional or quality requirements, logic itself does not need any separation into functions. That means as soon as functions are introduced into code, functional dependencies can be built which entail a new responsibility: Integration.

The beauty of Operations

In the beginning there was only logic. There were expressions, control statements, and some form of hardware access. And all this logic produced some required behavior.

Then the logic grew. It grew so large that it became hard to understand on a single level of abstraction.

Also in the growing logic patterns started to appear. So the question arose, why pattern code should be repeated multiple times?

Thus were invented subroutines (functions, procedures). They helped to make programming more productive. Patterns stashed into subroutines could be re-used quickly all over the code base. And they helped to make code easier to understand, because by calling a subroutine details could be folded away.

Before:

var x = a + ...;
var y = x * ...;
var z = y / ...;

After:

var x = a + ...;
var y = f(x);
var z = y / ...;

The change looks innocent. However it´s profound. It´s the birth of functional dependencies.

The logic transforming a etc. into z is not fully in place anymore but dependent on some function f(). There is more than one reason to change it:

  1. When the calculation of x or z changes.
  2. Or when something in the subroutine changes in a way that affects dependent logic, e.g. the subroutine suddenly does not check for certain special cases anymore.

Even though the logic and the subroutine belong closely together they are not the same. They are two functional units each with a single responsibility. Except that it´s not true for the dependent functional unit which has two responsibilities now:

  1. Create some behavior through logic (Operation)
  2. Orchestrate calls to other functions (integration)

To avoid this conflation the IOSP suggest to bundle up logic in functions which do not call each other.

Subroutines are a great tool to make code easier to understand and quicker to produce. But let´s use them in a way so they don´t lead to a violation of the fundamental SRP.

Bundle logic up in functions which do not depend on each other. No self-made function should call any other self-made function.

  • That makes Operation functions easy to test. There are no functional dependencies that need to be mocked.
  • That will naturally lead to small and thus easy to understand functions. The reason: How many lines of logic can you write before you feel the urge to stash something away in a subroutine? My guess it´s after some 100 or 200 LOC max. But what if no functional dependencies are allowed? You´ll finish the subroutine and create another one.

That´s the beauty of Operations: they are naturally short and easy to test. And it´s easy to check, if a given function is an Operation.

The beauty of Integrations

Once you start mixing logic and functional dependencies code becomes hard to understand. It consists of different levels of abstraction. It might start with a couple of lines of logic, then something happens in another function, then logic again, then the next functional dependency - and on top this all is spread across several levels of nested control statements.

Let´s be honest: It´s madness. Madness we´re very, very used to, though. Which does not makes it less mad.

We´re burdening ourselves with cognitive dissonance. We´re bending our minds to follow such arbitrary distribution of logic. Why is some of it readily visible, why is some of it hidden? We´re building mental stacks following the train of control. We´re reversing our habitual reading direction: instead from top to bottom and from left to right, we pride ourselves to have learned to read from right to left or from inner levels of nesting to outer and from bottom to top. What a feat!

But this feat, I´d say, we should always subtitle with "Don´t try this at home!" It´s a feat to be performed on stage, but not in the hurry of every day work.

So let´s stop it!

Let´s try to write code just consisting of function calls. And I mean not just function calls, but also function calls in sequence, not nested function calls.

Don´t write

a(b(c(x)));

instead write

var y = c(x);
var z = b(y);
a(z);

Let´s try to tell a readily comprehensible story with out code. Here´s the story of converting CSV data into a table:

Developer A: First the data needs to be analyzed. Then the data gets formatted.

Developer B: What do you mean be "analyzing the data"?

Developer A: That´s simple. "Analysis" consists of parsing the CSV text and then finding out, what´s the maximum length of the values in each column.

Developer B: I see. Before you can rearrange the data, you need to break the whole chunk of CSV text up. But then... how exactly does the rearrangement work, the formatting?

Developer A: That´s straightforward. The records are formatted into an ASCII table - including the header. Also a separator line is build. And finally the separator is inserted into the ASCII table.

That´s the overall transformation process explained. There´s no logic detail in it, just sequences of what´s happening. It´s a map, not the terrain.

And like any story it can be told on different levels of abstraction.

High(est) level of abstraction:

Developer A: CSV data is transformed into an ASCII table.

Medium level of abstraction:

Developer A: First the data is analyzed, then it´s formatted.

Low level of abstraction:

Developer A: First the data is parsed, then the maximum length of values in each columns is determined, then the records are formatted into an ASCII table - including the header. At the same time a separator line is build. And finally the separator is inserted into the ASCII table.

Finally the bottom level of abstraction or no abstraction at all would be to list each step of logic. That wouldn´t be an abstract process anymore, but raw algorithm.

At the bottom it´s maximum detail, but it´s also the hardest to understand. So we should avoid as much as possible to dwell down there.

Without logic details we´re talking about Integration. Its beauty is the abstraction. Look at the code for the above story about CSV data transformation:

image

Each function is focused on Integration. Each function consists of an easy to understand sequence of function calls. Each function is small.

Compare to this to a pure Operation:

image

Now, which solution would you like to maintain?

Yes, Integration functions depend on others. But it´s not a functional dependency. Integration functions don´t contain logic, they don´t add "processing power" to the solution which could be functionally dependent. Their purpose is orthogonal to what logic does.

Integration functions are very naturally short since their building blocks (function calls) are small and it´s so cheap to create more, if it becomes hard to understand them.

Testing

Testing Operations is easy. They are not functionally dependent by definition. So there is no mocking needed. Just pass in some input and check the output.

Sometimes you have to setup state or make a resource available, but the scope you´re testing is still small. That´s because Operations cannot grow large. Once you start following the PoMO and IOSP you´ll see how the demand for a mock framework will diminish.

Testing Integrations is hard. They consist of all those function calls. A testing nightmare, right?

But in reality it´s not. Because you hardly ever test Integration functions. They are so simple, you check them by review, not automated test.

As long as all Operations are tested - which is easy - and the sequence of calls of Operations is correct in an Integration - which can be visually checked -, the Integration must be correct too.

But still... even if all Operations are correct and the Integration functions represent your Flow Design correctly the behavior of the whole can be unexpected. That´s because flows are just hypothesizes. You think a certain flow hierarchy with correct logic at the bottom will solve a problem. But you can be wrong.

So it´s of course necessary to test at least one Integration: the root of a 3D flow.

Interestingly that´s what TDD is about. TDD always starts with a root function and drives out logic details by adding tests. But TDD leaves it too your refactoring skills to produce a clean code structure.

Flow Design starts the other way round. It begins with a functional design of a solution - which is then translated into clean code. IOSP and PoMO guarantee that.

And you can test the resulting code at any level you like. Automated tests for the root Integration are a must. But during implementation of the Operation functions I also write tests for them, even if it´s private functions - which I throw away at the end. I call those "scaffolding tests". For more on this approach see my book "Informed TDD".

Stratified Design

You´re familiar with layered design: presentation layer, business logic layer, data access layer etc. Such layered design, though, is different from 3D flows.

In a layered design there is no concept of abstraction. A presentation layer is not on a higher or lower level of abstraction compared to the business logic layer or the data access layer. Only the combination of all layers forms a whole.

That´s different for abstractions. On each level of abstraction the building blocks form the whole. A layered design thus describes a solution on just one level of abstraction.

Contrast this with the 3D Flow Design for the CSV data transformation. The whole solution is described on the highes level of abstraction by Format(). One functional unit to solve it all.

On the next lower level of abstraction the whole solution is described by Analyze() + Format_as_ASCII_table().

On the next lower level of abstraction the whole solution is described by Parse() + Determine_col_widths() + Format_records() + Format_separator() + Build_table().

Below that it´s the level of logic. No abstraction anymore, only raw detail.

How do you call those levels of abstraction? They are not layers. But just "level" would be too general.

To me they look like what Abelson/Sussman called "stratum" when they talked about "stratified design".

Each stratum solves the whole problem - but in increasing detail the deeper you dig into the flow hierarchy. Each stratum consists of a Domain Specific Language (DSL) on a certain level of abstraction - and always above the logic statements of a particular programming language.

Fortunately these DSLs don´t need to be build using special tools. Their syntax is so simple just about any programing language (with functions as first class data structures) will do. The meta syntax/semantics for all such DSLs is defined by IOSP and PoMO. They are always data flow languages with just domain specific processing steps.

Here´s another scenario:

An application displays CSV data files as ASCII tables in a page-wise manner. When it´s started it asks for a file name and then shows the first page.

Here´s a 3D Flow Design for this (see the accompanying Git repository for an implementation). See how the solution to the former problem now is part of the larger solution?

image

Vertically it´s strata put on top of each other. The deeper you go the more detail is revealed.

At the same time, though, there are the elements of a layered design. They stretch horizontally.

Colors denote responsibilities:

  • Integrations are white,
  • presentation layer Operations are green (Ask for filename, Display table),
  • data access layer Operations are orange (Load data),
  • business logic Operations are light blue (all else).

In stratified Flow Design, though, functional units of different layers are not depending on each other. Thus layering loses its meaningfulness. It´s an obsolete concept. What remains, of course, is the application of the SRP. User interaction is different from file access or table formatting. Hence there need to be distinct function units for these aspects/responsibilities.

In closing

The quest for readable code and small functions can come to an end. Both can be achieved by following two simple principles: the Principle of Mutual Oblivion (PoMO) and the Integration Operation Segregation Principle (IOSP).

That´s true for greenfield code where you might start with a Flow Design. But it´s also true for brownfield code. Without a design look at a function and see if it´s an Operation or an Integration. Mostly you´ll find it´s a hybrid. That means you should refactor it according to PoMO and IOSP. Clean up by making it an Integration and pushing down any logic into lower level functions. Then repeat the process for all functions integrated.

I suggest you try this with a code kata. Do the bowling game kata or roman numerals or whatever. Use TDD first if you like. But in the end apply PoMO and IOSP rigorously.

In the beginning you´ll be tempted to keep just a few control statements in Integration functions. Don´t! Push them down into Operations. Yes, this will mean you´ll get function units with several outputs. But that´s ok. You know how to translate them into code using continuations or events.

Even if the resulting integration might look a bit awkward do it. You´ll get used to it. Like you got used to reversing your reading direction for nested function calls. But this time you´re getting used to a clean way of writing code ;-) That´s like getting sober. Finally.

Organizing code according to PoMO and IOSP is the only way to scale readability and understandability. We need abstractions, but we need them to be of a certain form. They need to be clean. That´s what IOSP does by introducing two fundamental domain independent responsibilities: Integration and Operation.

The beauty of this is, you can check for conformance to the SRP without even understanding the domain. Integration and Operation are structural responsibilities - like containing data ist. You can review the code of any of your colleagues to help them clean it up.


  1. Remember my definition of logic: it´s expressions, control statements and API-calls (which often stands for hardware access of some kind).

  2. I know, you´ve tried hard for years to keep the number of lines in your functions low. Nevertheless there are these monster functions of 5,000 LOC in your code base (and you´ve heard about 100,000 LOC classes in other projects). Despite all good intentions it just happens. At least that´s the code reality in many projects I´ve seen. But fear not! You´re about to learn how to keep all your functions small. Guaranteed. I promise. Just follow two principles. One you already know: the Principle of Mutual Oblivion.