The Architect´s Napkin

Software Architecture on the Back of a Napkin
posts - 65 , comments - 171 , trackbacks - 0

My Links

News

Archives

Thursday, May 21, 2015

More than one order matters

Order matters. In real life when you’re in a library or a city or in your kitchen. And in software development. Order, the right order, helps understanding. Where there is order orientation is easier. Compare a city like Mannheim, Germany:

image

…to Algier’s traditional quarters, the Casbah:

image

Where, do you think, it would be easier for you to find the house of a friend?

Order is helpful, even important. But often order is hard to create and/or to maintain. Why’s that? What’s order anyway?

Order’s definition

If you look at Wikipedia it seems order is quite an elusive thing. There is no single page just defining the order I’m talking about and you very likely have in your mind.

Yes, order is about “an arrangement of items”. But it’s more than a “sequence”. The Free Dictionary defines it as:

“A condition of logical or comprehensible arrangement among the separate elements of a group.”

or

“A condition of methodical or prescribed arrangement among component parts such that proper functioning or appearance is achieved.”

That clears it up, doesn’t it? :-)

No, not really, I guess. So let’s dissect the definitions:

  • “arrangement”: Order is about “things” (elements, items, “component parts”) and their relations. Arranging “things” this way or that changes their relations with each other.
  • “proper functioning”: Order has a purpose. Elements in order serve the purpose better than elements in disorder. Thus order is the result of a conscious effort aiming at creating something “useful” through order.
  • “comprehensible”: Ordering items leaves them in a state which is easier to comprehend than the state they were in before. Reasoning about the items and their relations becomes easier. This requires some logical system consisting of rules and principles, though.

To sum it up I’d say “order” means:

A conscious arrangement of items according to a logical system to best serve a specific purpose.

This explains why Robert C. Martin says in his article “Does Organization Matter”:

“The organization structure that works for a thousand line program, does not work for a ten thousand line program. And the organization structure that works for a ten thousand line program does not work for a hundred thousand line program.”

He speaks of three different purposes which a single order cannot address. A 1K LOC program very likely serves a different purpose than a 10K LOC program or a 100K LOC program. That’s why they need different orders.

And in his article “Screaming Architecture” he argues that even two programs both with the same number of LOC should have different orders if their domain differs, because the order (structure) should mirror the purpose.

Order’s purpose

Since creating order is a conscious effort, it not only needs to enable some purpose (e.g. a motor pulling some weight or a software enabling online shopping), though. It also needs to serve its creators. Creators (and observers) need to understand it, otherwise it would be hard to create it in the first place or change it later on for adaption to a changed purpose (requirements).

An intelligible order is an order where cause and effect relationships can be determined. In software development this means questions like “If I change this here how does that affect the behavior of the software?” and “If I change this here how does that affect the evolvability of the software?” become easier to answer with more/better order.

Take these two arrangements of Mikado sticks as an example: For which arrangement do you find it easier to determine what happens when you move a stick?

image

To me, the purpose of order itself is to arrange items not just in any way to serve a “behavioral purpose”, but to arrange them in a way so that complexity is minimized.[1]

That’s why the “logical system” is so important behind any order. Because a “logical system” is an abstraction. It lifts a huge number of trees onto the level of a forrest. Causal relationships are easier to recognize if there are less details to take into account.

Multiple orders

By now it should be clear why order is important in general and in software development in particular. Without order, i.e. a conscious arrangement of code items following some “logical system”, a software’s behavior is pretty much accidental. And reasoning about it in order to fix a bug or add a feature might be near to impossible.

However, it should also be clear, a software system cannot be ordered in just one way.

Take a kitchen for example. I’m sure you appreciate an orderly kitchen with cupboards and drawers as containers for silverware and dishes.

image

The order is high - or to venture into the realm of physics: the entropy is low - because if you know the location of a single fork or plate it’s easy to reason about the location of other pieces of your household. There is a conceptual system behind how all this is arranged in a kitchen. And if not in your own kitchen then at least in a restaurant’s kitchen ;-)

But why arrange kitchen items in an order? Why not pile them up in a corner of the kitchen? That might damage them, it’s bad for hygiene - but above all it makes it harder to find an item when you need it. The specific purpose of the order in a kitchen is to enable efficient cooking and serving.

Now look at this picture:

image

That’s items taken from a kitchen - but in a different order. The same items can be arranged in different orders for different purposes. Here the purpose is to make eating easier.

And finally this:

image

Again items taken from a kitchen - but in yet another order.[2] What’s the purpose here? Maybe to ensure all items are cleaned best or maybe maximum load (with less cleanliness).

Anyway, as you see, the same items can be arranged in multiple orders to serve multiple purposes.

And I guess it also has become obvious that order is relative. Order is in the eye of the beholder. You need to at least understand the “logical system” behind an order to appreciate an arrangement. That’s why - if you’re like me - you should visit a gallery of modern art only with some kind of guide. Otherwise you might see chaos where there indeed is order ;-)

Multiple software orders

Now finally on to software. What kind of order does software need? And what is it that needs to be consciously arranged?

The last question is easy to answer. It’s logic and data that needs to be arranged. With logic being programming language statements producing behavior, i.e. transformations, control statements, and hardware access/API calls.

But to what purpose should these language constructs be put in an order?

I suggest to order code to fulfill some requirements. Creating order takes time, i.e. it costs money. So it better serves the source of money, the customer. That means it should be of some use to her. What’s of use to a customer is stated in a requirements document.

Requirements can be divided into two broad and obvious categories:

  • Functional requirements
  • Non-functional requirements

But that’s not the only way to slice requirements. Here’s another perspective:

  • Runtime or behavioral requirements
  • Design time or evolvability requirements

Both categorizations can be brought together in a triple:

  • Functional runtime requirements; example: an online shop should let you search products.
  • Non-functional runtime requirements; example: product search in an online shop should be so fast it shows results while you’re typing.
  • Non-functional design time requirements; example: the online shop should be (quickly) adaptable to all sorts of unforeseen requirements over the next three years.

Maybe you come up with even more requirement categories. But even with just these three at hand the question is: How do you create separate orders for each of the categories?

Modules

The order creating the most heated discussions is the one pertaining to non-functional design time requirements. When you do object-oriented programming you do it for this reason.

Classes are modules like libraries and components and functions. Modularization thus is the art of imposing an order on code to increase developer productivity and code evolvability.

Here’s a class design as an example for a module order from a recent article of mine:

image

Hosts

The order most desired is a different one, though. It’s the order needed to deliver on non-functional runtime requirements (or efficiencies) like performance, scalability, security etc.

How should logic be arranged to make a system scalable or responsive? And how do you call the containers for code to pack-up logic in for that purpose? I call them hosts as opposed to modules.

Hosts run code on a thread or in a process or on a machine or in a network. It’s the same code as arranged in a module order - but now arranged differently. Like with silverware and plates. Two purposes, two orders.

Here is an actor diagram from the same recent article as an example of a host order:

image

Don’t be surprised to see classes in this diagram. Classes are used to manifest some hosts in code.

Flows

But wait, there is more! What about functional requirements? What’s the order for them? Interestingly there is hardly any talk about order to fulfill functional runtime requirements.

There are some diagrams to depict functional order, e.g. flow chart, Nassi-Shneiderman diagram, sequence diagram, activity diagram. But their use is comparatively rare and not very systematic.

But functionality needs order like performance or evolvability. That’s why I suggest to become more conscious about it and to build skills to design it. The tool for that to me is Flow Design. Yes, I believe functional order is best expressed by declarative flows, that is data flows.

Data flows describe functional behavior as a series of processing steps connected by messaging.

Finally here is an example data flow from the mentioned article as an example of functional order:

image

The functional units you see are encapsulated by modules and hosts.

Designing orders

The orders for functionality, efficiency, and evolvability are orthogonal to each other. You can change the module order without changing the functional order. You can change the efficiency order without changing the module order. At least in principle, at least to a certain extend.

Because the orders intersect. They intersect at the function level. Functions are the smallest modules and the smallest units of the functional order and the smallest containers to contain logic for some efficiency requirement to be distributed across threads and operating system processes etc.

  1. Since the first requirement to fulfill is the functional requirement, functional order is the first to establish. Start by defining behavioral processes. You need to give the input-output transformation an orderly structure. They will be running whenever a user triggers a software’s behavior.
  2. Once functional units have been arranged in data processing flows start to arrange them in a host order. Distribute them in a way so they fulfill efficiency requirements, e.g. by running them on their own threads or even devices.
  3. Finally arrange the functional units in a module order. Distribute them in a way to maximize evolvability - without sacrificing any of the host order requirements.

Although I listed the order design steps in this order and recommend to advance in this way through software design, sooner or later you’ll change the sequence. Depending on the size of the project you might want to start with a coarse grained host order, then continue with the functional order, and determine modules last.

Anyway, it’s up to you. Just don’t forget to give your code an order in all relevant dimensions. There is always more than one order to code. And all of them matter.


  1. My personal definition of complexity is simple: A system is complex if cause and effect relationships are not clear. Your TV set is not complex; it’s perfectly clear to you what happens when you press a button on its remote control. But an airplane is a complex system. Unless you’re a pilot it’s not clear what’s gonna happen when you press a button or turn a knob on the dashboard in a cockpit. On the other hand a system is only complicated if cause and effect relationships are clear or can be found with some reasonable effort. Now think about relationships with friends and your spouse: Are your social systems complicated or complex? ;-)

  2. Believe me, even if you sometimes might think there is no order to how items are place in your friend’s dishwasher, there is an order. Just try to add another item to an already fully loaded dishwasher and your friend will certainly tell you you’re doing it wrong ;-)

Posted On Thursday, May 21, 2015 3:42 PM | Comments (2) |

Tuesday, May 12, 2015

Actors in a IODA Architecture by Example

The IODA Architecture is supposed to make software design easier and improve code readability for about any problem domain. I’ve tried to show an example of that in my previous article. It was a larger scenario than the usual code kata, but still it was a problem to be solved by synchronous programming.

In this article I want to move beyond that and tackle a problem where multiple threads need to be employed. Here’s the challenge:

Write an application to work as an alarm clock. It displays the current time and the user can set a wakeup time. Once the wakeup time is reached the application sounds an alarm.

Requirements analysis

This scenario needs some user interface design. How should the user interact with the program’s body? How about this:

image

When the user switches the alarm on, the remaining time until the wakeup time will be displayed. Both current time and remaining time will constantly (every second) be updated.

The user can switch off the count down at any time. The remaining time will then disappear.

When the wakeup time is reached a WAV or MP3 file should be played by the application.

Trying to bridge the requirements-logic-gap I see there is just a single dialog to this application, and within the dialog there are two interactions. Let’s call them “switch on” and “switch off”.

However… behavior is not only triggered by a user putting the switch left or right. The application is supposed to also behave in some way on the tick of a clock. So there is another “invisible user” interacting with the application. It’s the system clock.

From this I derive a picture of the whole application.

image

The system-environment diagram shows very application specific behavioral aspects separated into their own modules. I don’t know yet how those modules will be working together, but it’s clear to me at least these modules should be present. The portals and the provider each encapsulate an API. And the domain is the home of the very alarm clock logic.

Although “module” sounds technical this to me is still analysis. Because these aspects can readily be found in the requirements. They are so tangible I could and should talk about them with a customer.

But please note: According to the IODA Architecture these behavioral modules are not related to each other in any way. There are no functional dependencies between them whatsoever.

Finally one last step in the requirements analysis phase: finding features. Interaction behavior usually is not monolithic. So what behavioral aspects can be found in each interaction?

  • System clock tick
    • Update display of current time
    • Update display of remaining time while alarm is switched on
    • Check if wakeup time has been reached
    • Sound alarm is wakeup time has been reached
    • Switch off checking for wakeup time if wakeup time has been reached
  • Switch on the alarm
    • Switch on checking for the wakeup time
  • Switch off the alarm
    • Switch off checking for the wakeup time

To me Agile Architecture means, such increments have a 1:1 correspondence to programming language structures. Dialogs become classes, interactions become functions, and features also become functions. So what the analysis phase has brought me is a first rough sketch of my code. It will consist of at least 4 classes (dialog portal, system clock portal, alarm bell provider, domain). And there will be at least 3+7=10 functions for all the interactions and features.

I guess, this is all I need to enter the next phase: designing the solution, i.e. describing the behavior visible on the outside by building it from smaller behaviors in a declarative way.

Solution design

My initial sketch of the data flows in the application looks like this:

image

It took only maybe 5 minutes to come up with it. I like to draw flow designs with pen and paper. It’s flexible and simple. No other tools needed.

But I’d understand if you have a hard time deciphering my handwriting ;-) So let me go through the parts step by step and redraw them in a legible way.

Let’s start with the most interesting interaction: the system clock tick.

image

For every tick of the system clock it is checked, if the wakeup time has been reached. Of course this check is only done while the alarm is switched on (see status).

Also, while the alarm is switched on, the remaining time until the wakeup time is calculated. Both current time and remaining time flow to the UI to be displayed.

In case of the wakeup time having been reached this is flagged to the UI and the alarm bell is rung.

After this interaction the rest is easy. Let’s do start and stop in one diagram:

image

Starting the alarm clock means registering the wakeup time and setting the status to “active”. Stopping it means setting the status to “inactive”.

Taking multithreading into account

That’s an easy design - but unfortunately it would hardly work if implemented just like this. The reason: there are multiple threads at work here. This would at least cause difficulties for displaying the current/remaining time. They flow into the UI from a non-UI thread. You can see this in the following diagram: a red data flow arrow is connected to blue box.

image

Interestingly the same happens in Check wakeup time: the functional unit accesses the wakeup time and the status from the red thread - but this state is blue, because it first got set by Start which gets its data from the UI.

Multiple threads reading and writing the same resource does not seem like a good idea. So what can we do about it?

To solve the problem of data flowing into the UI from a non-UI thread it’s easy to insert a synchronization point. This functional unit just takes data coming in on some thread and passes it on on another thread, the UI thread.

image

Technically this is no problem in .NET with a SynchronizationContext.

But how to protect access to the common state of Start and Check wakeup time reached and Stop from multiple threads? That could be done with a lock.

For a long time I’d have gone down that road - but not anymore ;-) I’ve become a fan of the actor model and its .NET implementation Akka.NET.

Before I introduce actors into the design, though, some class design is in order after the flow design.

Class design

So far the design contains only “functional units” (FU) as processing steps in a data flow. Those FUs will be implemented as functions. But which classes should these functions be put in?

This is easy, I’d say. Have a look:

image

UI, clock and alarm bell all get their own classes. I guess that’s a pretty obvious decision.

And Watchdog? That class encapsulates the state shared by the remaining functional units. Shared state to me is a “hard aspect”. It calls for being wrapped in a module of its own.

Actor design

From the threading point of view the class situation looks like this:

image

Two classes need their own threads: the UI runs on the UI thread, the clock on some system thread.

The Watchdog is accessed by multiple threads and needs to protect its inner state.

And the Alarmbell pretty much does not care. It receives data on the clock thread, but does not need to do its job on that thread. Depending on how the alarm bell gets implemented it can run on the clock thread - or maybe gets its own for even more decoupling.

Actors help to implement this without resorting to special synchronization mechanisms like SynchronizationContext or locks. In this case I just need to encapsulate each class in its own actor:

image

Between actors flow messages. They are received by the actors’ “inboxes” and worked upon when a previous message has been processed. All actors work on different threads. But each actor just on one (although that might be a different one for each message). That way there cannot be any conflicts when accessing state. Actors protect any state by sequential message processing.

Message design

Data flowing between two functional units in a Flow Design often does not need a special type. It’s clear what it is and what’s supposed to happen with it. A single processing step representing an activity produces data which is consumed by a single processing step representing an activity.

image

With actors that’s different. Actors are more like nouns than verbs. They represent entities rather than activities. That’s why they are called actors. Hence actors often receive many different messages to process - but all in a single “inbox”.

In order to find the right message handler it helps to send messages of very concrete types. That could be commands or events. Commands describe, what’s supposed to happen. Events describe what has happened.

Let me take a subset of the actor flow design as an example:

image

Each actor has a single entry point into which all messages flow.

The UI actor outputs/sends commands because it is the one relating the user intentions to the rest of the program.

The watchdog actor on the other hand outputs/sends events informing other actors about what happened.

Commands and events are “implicit” in the original flow design. They now have to be made explicit in order to have the actors communicate clearly with each other. See the GitHub repo with the implementation for a list of them.

Implementation

I implemented the application in two phases:

  • First I did it without actors. There were only the classes and synchronization with the UI thread. Not even a lock around the watchdog state. This I did not to get it 100% right, but to see if my idea of an alarm clock flew at all. It was much more than a walking skeleton - but not really the fully fleshed application I wanted. You find the sources for this version here.
  • Only in a second step I wrapped actors around the classes. This proved very easy. I just took what was already working and put in another set of containers. Functions and classes are modules; modules are containers to produce evolvability and productivity. Actors on the other hand are host; that’s how I call containers to produce some efficiency, e.g. a non-functional requirement like performance, scalability, or responsiveness. The latter is required for this application. The UI must stay responsive to be able to start/stop the alarm clock at any time. You can find the sources for this version here.

The implementation did not pose any challenges, except one: Getting the alarm bell running on OSX. I’m working with C#/Mono on a Mac and all’s well including doing graphical user interfaces with WinForms. (WinForms GUI’s on a Mac don’t look that nice, but they serve the basic purpose.) The simplest way to play sound files in .NET/Mono is to use the System.Media.SoundPlayer class. But it simply would not work with a .WAV file on OSX. Bummer. So I had to switch to an out of process call to the OSX utility afplay. This led to an interface for the alarm bell - IAlarmbell - and two implementations: AlarmbellOSX and AlarmbellWin.

But that’s really a minor point. Most important is that switching from “regular” multithreading to actors worked like a charm.

The first actor I wrote was the clock. I call it an actor even though it’s not an Akka.NET actor. Rather it’s a simple class doing its work on its own thread: every second it publishes the current time via a C# event.

image

I needed it for the first phase and saw no reason to wrap a real actor around it in the second phase.

The other actors I wrapped around an existing class. For example the Watchdog class to check, if the wakeup time has been reached:

image

Each of the “bubbles” of the design becomes a function. Input flows into these “bubbles” as function parameters. But the output of Check flows out via C# events. I could have implemented it with continuations instead, but wiring up the initial classes in phase #1 was easier with events. Anyway… I hope you agree, this looks pretty straightforward and clean.

This class is free of any multithreading concerns. I can test it easily.

Enter the actor: The WatchdogActor takes a Watchdog instance and wraps it so it becomes thread-safe.

image

See how the actor itself does not contain domain logic. It just wires its “inbox” to the “input ports” of the domain class:

Receive<StartCommand> (cmd => dog.Start_watching_for (cmd.WakeupTime));

(Please see the Akka.NET documentation on how receiving and sending messages with actors works. Or even better: Join the Akka.NET Bootcamp to get your hands on this awesome framework.)

The same happens for the "output ports" of the domain class. The actor registers handlers for each event, wraps the event data into an event object, and sends it on...

dog.OnRemainingTime += remainingTime => {
 
var e = new RemainingTimeEvent{RemainingTime = remainingTime};
  onEvent.Tell(e, self);
};

The domain class does not know what’s done with its output downstream in a data flow. Neither does the actor know what’s done with the event objects it outputs. It’s a matter of the integration in Program.Main() to determine who’s going to listen to whom:

image

Actors are created through a factory, e.g. sys.ActorOf (dogActorProps, “WatchdogActor”). The factory gets what it needs to produce a Watchdog via a Props structure:

Props.Create<WatchdogActor> (dog, new[]{ "DlgActor", "AlarmbellActor" });
...
Props.Create<DlgActor> (dlg, dogActor)

The parameters of Create() can be the ones passed on to the constructor. In this case that’s an instance of the domain class, e.g. dog for Watchdog.

In addition I pass in a list of actors who are recipients of the actor’s output. I do that with a list of actor names. This list then feeds an ActorMultiRef which is used to broadcast messages to multiple actors:

var onEvent = new ActorMultiRef (Context, eventReceivers);
...
onEvent.Tell(e, self);

Akka.NET does not seem to support “multicast events” in any other way. Routers seem too heavyweight for that purpose. So I created my own small multicast actor reference. It looks up the concrete actors by name. That way I can “forward reference” actors not yet instantiated. This lets me integrate actors in a very easy way and as flexibly as ordinary objects.

Summary

The next diagram shows the class hierarchy of the solution:

image

As you can see there are two strata of integration and - again - only one for operations. Only the Operation-classes contain logic. Neither Integration- nor Data-classes contain any. There are no functional dependencies in this solution.

The actors solve an efficiency problem as hosts - but at the same time work as modules integrating domain behavior. This integration is very simple, since there is a 1:1 relationship between actors and the “workhorses” they “ride”. But still… The resulting class diagram is very, very simple. And the “workhorses” (operations) are easy to test.

My bottom line on this project is: Actors fit nicely with the IODA Architecture and Flow Design. And they make solving problems with multiple threads easier.

To me actors look like coarse grained modules. They are a different kind of classes so to speak. Their USP is the much more rigorous encapsulation of internals. They feel quite a lot like objects as Alan Kay envisioned them. They are somewhere in the middle between class and µService with their asynchronicity and message based, but platform dependent contracts.

I guess, I’ll use more of them in the future :-) Not for technology’s sake, but as a means to encapsulate logic even better than with classes - and at the same time improve on performance and scalability.

Posted On Tuesday, May 12, 2015 6:29 PM | Comments (5) |

Saturday, May 2, 2015

IODA Architecture by Example

The IODA Architecture to me is fundamental. But as many fundamental notions it might be a bit abstract. That's why I want to provide an example of how to design and implement software based on it. I keep things easy, but not trivial. So here is a scenario:

Build an application that translates a roman number entered by the user into an arabic number and vice versa.

This sounds easy, doesn't it. It's more than the code kata Roman Numerals requires, but not much.

Requirements analysis

Before I start with the design, a short analysis seems to be in order.

The requirements ask for an application. That means the user should be able to start a program which converts numbers. But how should the user interact with the program? Since the requirements don't tell me much about it I assume the simplest approach: a console application. It could work like this:

$ convertroman XIV
14
$ convertroman 42
XLII
$

Roman number conversion is no rocket science. I won't go into it here. Look it up on Wikipedia, if you like.

What about validation? How should the application react to negative arabic numbers or invalid characters in a roman number? The requirements are silent about this. So I assume just very, very basic validation.

The main purpose of this analysis is clarification of how data gets into the program and how output is presented to the user.

Solution design

The IODA Architecture does not prescribe any behavioral responsibilities. So what can I start with? I could dig into the requirements and identify candidate classes, if I was doing traditional object orientation. But I'm not.

Objects - at least the traditional ones - are secondary. What is primary is behavior. What's supposed to happen? That's the driving question.

At the beginning I know very little. I just know, "everything" needs to happen upon a trigger from the user.

image

But then, when I think about it, I do know a bit already. It's just a pattern:

  1. The input data has to be read. It's passed in via the command line.
  2. The data has to be converted.
  3. The conversion result has to be displayed.

Flow Design

The overall behavior is made up of "parts"; there are distinct processing steps. It can be refined. The current functional unit "convert roman" becomes the root of an integration hierarchy.

image

Now I repeat this: For every functional unit I ask myself, from what even smaller processing steps it could be made up of. If I truly understand the problem I'm sure I'll be able to name at least a few such sub-tasks.

  • How complicated is it to read the input number from the command line? It's trivial. No refinement is needed.
  • How much effort is it to display the result? It's trivial, too. No refinement needed.
  • How does the conversion work? That's not trivial. Refinement seems necessary.

Conversion has at least two sides to it: converting a roman number to its arabic equivalent, and converting an arabic number to its roman equivalent.

Also which conversion to choose needs to be decided.

And how about the simple validation?

Let's put these behavioral pieces together. Here's the "domain process" to do the conversion:

image

In the former design "convert" was a leaf, an operation. But now it has been opened up and become an integration.

Again I repeat asking myself for every functional unit how difficult it seems to implement it. If I clearly see an implementation before my mental eye, I won't refine it further during design. Otherwise I need to do a bit more thinking. How could the partial behavior possibly be accomplished as a "domain process"?

To me such a refinement seems to be warranted for both conversions. They are at the heart of the application. That's really the core domain of the program.

Here is the exemplary decomposition of "convert from roman": My approach to solve the problem consists of three processing steps.

  • Each roman digit of the roman number is translated into its value, e.g. "XVI" -> [10, 5, 1].
  • The values are summed to calculate the arabic number, e.g. [10, 5, 1] -> 16.
  • In case of a smaller value standing in front of a larger value the smaller value is negated, e.g. [10, 1, 5] -> [10, -1, 5]. I call this "applying the subtraction rule".

Correctly wired together the "domain process" consisting of these three steps looks like this:

image

If you, like try the same for the conversion of arabic to roman numbers ;-)

Should the new functional units be refined further? I don't think so. I feel confident to just write down the code. It's pretty obvious, isn't it?

So much for the happy day case. But what about a rainy day with invalid input? What to do about it? I think some error output would be nice. I'll extend the data flows from above like this:

image

It's never too late to improve a design ;-)

Let's step back: What is this I've designed so far? It's data flows; I call it Flow Design. It's a declarative description of what's supposed to happen to solve a problem (which always is some transformation of input into output and side effects).

At this point I don't have a clue how the actual logic in each of the operations will look like. That's the imperative part. It's implementation details.

But my feeling is ;-), if all operations are implemented correctly, then the integration of them in the depicted manner will lead to a correctly working program.

Here's the complete final Flow Design for the program with operations marked with dark color:

image

Data Design

So much for the I(ntegration) and O(peration) strata of the IODA Architecture. But what about the D as in Data?

Of course it's already there. Data is flowing between the operations. Here's a section of the Flow Design across levels of abstraction with focus on the data:

image

Due to the simplicity of the scenario the data types are all primitive. It's strings and integers. Maybe, yes, maybe it would be nice to introduce specific types for domain concepts like roman number and arabic number. But right now I don't see much value in that; it would feel kinda artificial/overengineered. Especially since I'm going to use C# as the implementation language. New types are comparatively expensive using that syntax; I'd be more inclined to define types like that if I was using F#.

Anyway, whether the scenario calls for its own data types or not, this does not change the basic architecture: operations depend/work on data. If there were any specific data structures they'd reside beneath the operation stratum.

Class Design

The "bubbles" you've seen so far will be translated to functions in a systematic way. Wait for the implementation...

Functions are modules. And I think each of the bubbles designed so far has a pretty narrow responsibility. It focuses on a single aspect of the requirements. See my article on the Single Responsibility Principle (SRP) for details.

But as important as it is to focus the responsibility of each function (aka bubble), there are more module levels available. I should use them to form higher level responsibilities.

Also, any functions that might be created from the bubbles designed need a class as its home.

Before moving on to the implementation I need some clarity about what classes the program should consist of.

Notice the difference between the usual OOD/P approach. I haven't thought about classes first, but about behavior/functions. Behavior is primary, classes are secondary.

So I did not have to come up with any "candidate classes". My classes are the real ones right away. And I know exactly which functions go where. Because I abstract classes from patterns I see in the Flow Design.

Here's my idea of how to group bubbles into classes:

image

Some of it might be obvious for you, some not. So let me explain my thinking:

  • Providers: Using an API is a "hard" aspect in any software. It should be isolated into specific modules. Usually I'd say each API should be encapsulated by its own class - but in this scenario there is so little of it, I guess a single class will do.
  • FromRomanConversion and ToRomanConversion: Doing the actual conversion is an aspect of its own. Since there are two directions of conversion a module for each seems to be in order.
  • RomanConversions: This class belongs to the coarse grained domain aspect of the application. It contains "helper functions" not actually concerned with the transformations.
  • Body: This class represents the overall functionality - but without the interaction with the user. It's like an internal API to what the program is supposed to do.1
  • Head: The head is responsible for triggering body behavior. It integrates the body with input from and output to the user.

Interestingly the class diagram follows the IODA Architecture like the bubbly Flow Design:

image

Only the data stratum is missing. But that's just because the scenario is so simple.

Interfaces and Static Classes

Class design is more than determining which classes there are. It's also about how the services of those classes should be accessible.

Question #1: Can any classes be static? I'd say that's a legitimate question; not all classes need to be instantiated. This only makes sense for classes with data or those which should be replaceable or into which replacements (mocks etc.) should be injected.

Looking at the class design for this application I think, RomanConversions, ToRomanConversion, and FromRomanConversion can be static. There is no state involved, no APIs need to be replaced for testing purposes.

This is different for Providers. It encapsulates APIs which might make it hard to test its users like Head. An interface seems in order or maybe even two: one for the CLI part and one for the console output part of the class. Why not apply the Interface Segregation Principle (ISP)?

From that follows Head cannot be made static. It needs to get instances of the provider interfaces injected, since it statically depends on the interfaces and dynamically on some implementation of them.

Also Body should be described by an interface. That would make it easy to test the head in isolation.

Library Design

Aspects are hierarchical so they should be mapped to a hierarchy of modules. Functional units of the Flow Design have been aggregated into classes. But what about the classes?

The next level of modules after classes is libraries. At least in my thinking.

Also libraries are a means of organizing code in many IDEs under the name "project" (eg. Visual Studio) or "module" (eg. IntelliJ).

Again I don't start with any library candidates, but rather let the classes inspire me. What should be grouped into a library? Here's my answer:

image

  • RomanConversions contains all classes of the business domain.
  • Providers is the home for all classes dealing with APIs.
  • Head is the top level integrator, the entry point of the application.
  • Body does the integration of the "backend". That's where behavior is created.

Component Design

The next level in the module hierarchy are components. A component consists of one or more libraries and is described by a separate platform specific contract. This enables a more "industrial" style of software production, since work on different components can be done in parallel, even if they depend on each other at runtime.

The current scenario does not really call for components. I'm working on it alone. But still... I'll make Providers and Body components, since there are interfaces and possibly several implementations over time. It's not much effort, so there will be another library for the contracts.

Implementation

Implementing the design is easy. Almost boring ;-) There are clear rules how functional units of Flow Designs should be translated into functions. It took me some 90 minutes to write down the code including all tests.

image

Test-first coding wasn't necessary since all operation functions are so small. Also the design was already there so I did not need tests to drive it. The tests I set up are there to avoid regressions.

You can view the sources on GitHub.

When looking at the code there should not be any surprises. The project layout follows the component/library design.

image

The dependencies between the libraries/classes are as planned.

image

The code has a IODA architecture on all module levels from functions to components.2

One thing, however, is noteworthy, I guess. How did I translate the bifurcations in the Flow Design, i.e. where the flow splits in two? There are two outputs to "convert", "determine number type", and the validations.

When you look at the following code you might need a minute to understand what's happening:

image

But actually it's simple. Just read the function calls from top to bottom and left to right. Indentations denote different paths in the flow.

Here's the topmost flow side by side with its implementation:

image

Each bubble translates to a function call. If there is only one output, the function returns a value. If there is more than one output, the function passes on a value using a continuation (function pointer). Those continuations are what makes the code a bit hard to read. But only at first. It's unusual, not wrong. You just have been trained for so long to read nested function calls from right to left and inside-out; but once you get used to continuations you'll come to appreciate how easy it is again to read code.

Elements of Functional Programming are used to translate the declarative Flow Design into OO language code.

But that's only details. My main point was to show you, that designing a IODA Architecture for an application is not difficult. And implementing the design is not difficult either.

The universal aspects of integration, operation, and data (admittedly not much in this case) have been separated. And orthogonal to that behavioral aspects have been identified and encapsulated into modules on different levels.


  1. This root class could also be called Application or maybe viewed as a use case and named accordingly.

  2. There are slight exceptions. Although the classes FromRomanConversion and ToRomanConversion are operations from the point of view of the class design, they contain both integration functions and operation functions. As long as there is a large imbalance between the two aspects that’s ok. If the number of integration functions would grow in such a hybrid class, though, it would hint at splitting it into dedicated integration and operation classes.

Posted On Saturday, May 2, 2015 2:19 PM | Comments (0) |

Wednesday, April 29, 2015

The IODA Architecture

The common architectural patterns are all pretty much the same. Layered architecture, MVC, Hexagonal architecture, Onion Architecture, Clean architecture… they all do two things:

  • Define domains of responsibility
  • Put functional dependencies in order

Look at these depictions of some of the patterns:

image

What you find are shapes (rectangles, circles) with the responsibility to produce part of the required behavior of a software. All models forsee certain responsibilities like interaction with the user, persisting data, representing and working with domain data etc. Some patterns are more detailed, some are less. That way they are more or less applicable to concrete requirements.

And those shapes are explicitly or implicitly connected in service relationships. One shape uses another; one shape depends on another. Sometimes arrows denote these relationships, sometimes mere location (e.g. a shape on top of another means “top uses bottom”).

These uses-relationships are cleanly directed and there are no cycles.

This is all nice and well. It helped us along quite a while.

But again and again I’ve seen developers scratching their heads. What about stored procedures in a database? They implement business logic, but don’t reside in the Business Layer. What about JavaScript validation logic? It’s part of the Presentation Layer, but implements business logic. Or what if I don’t want to go down the DDD path with Domain Entities and Services? Maybe the application is too small for this.

Most of all, though, what boggles the mind of developers, as far as I can see, are the functional dependencies between all those responsibilities. It’s deep hierarchies which won’t go away by switching to IoC and DI.

Looking at a function on any level of this hierarchy is an exercise in archaeology. Put on your head lamp, get out the brush, be careful while you uncover mysterious code hieroglyphs of the past. There is so much to learn from those 5,000 lines of code… And there, look, another tunnel leading you to yet another level of code.

As useful as the architectural patterns are, they are only coarse grained and they leave us with a very crude idea of the fundamental structure of software. According to them, software is just a deep hierarchy of services with particular responsibilities calling other services calling other services etc. for help.

Thus the DI container has become the developers best friend. And his second best friend has become the mock framework.

This is bothering me.

Certainly every software system is made up of aspects which should be encapsulated in modules. There might be a UI aspect or a persistence aspect or a use case aspect or an encryption aspect or a reporting aspect etc. etc. They are all a matter of the Single Responsibility Principle, which I’ve already put under the microscope.

At this point I don’t want to argue with the established architectural patterns about the responsibilities they perceive as inevitable, be that Business Layer or Repository or Controller etc.

Let’s call them behavioral responsibilities. Because each responsibility is about a functional or non-functional aspect of the behavior the customer wants a software to show.

But what I question is, whether the fundamental notion of software being just a uniform hierarchy of services is useful. And service being some kind of behavioral responsibility to be used by another service.

image

Essentially there is no distinction. Look at the code in a function in a Presentation Layer or a Use Case or a Controller or a Respository. You won’t see a difference - except in behavioral purpose. All functions on all levels of the dependency tree contain logic and calls to other functions.

“That’s how it is. That’s the nature of software, isn’t it?”, you might say. But I beg to differ. This is just how we structure software today. It’s a mush. And that’s why there are functions with 500 or even 10,000 lines of code. There is no technical limit to how many LOC a function can have. And there is no binding rule about how many LOC it should have (except for some recommendations). It’s all up to us developers. So if push comes to shove a method just grows - one line at a time. Because it can.

This is what has been happening for the past 50 years. And it has been happening despite all well meant design patterns or architecture patterns.

Why’s that? My guess, it’s because all those patterns do not question the basic assumption of all levels of dependency being created equal.

Patterns so far have been concerned with defining certain behavioral responsibilities and nicely ordered dependencies between them. That’s all.

So I suggest, if we don’t like the current state of codebase affairs, then we should try something new. Here’s what:

Formal responsibilities

I’d like to propose to search for the fundamental architecture of software in another dimension. The current architectural patterns are just evolutionary steps in the same dimension. So let’s step out of it. Let’s turn 90° and start moving down another path.

It’s one thing to search for responsibilities in the behavioral domain. That’s necessary, but also it is limited. Like the Flatlanders baffled by the effects of a 3D object passing through their 2D space, we’re again and agin baffled by the temptation to name some class “manager” or “coordinator”. It doesn’t sound right.

Locked in our current view of software design the need for managercoordinatorcontroller classes seems like a symptom of bad object-orientation.

But what if we broaden our view, step out of our dimension or behavioral responsibilities?

Let me introduce three formal responsibilities. I call them formal, because they are not concerned with creating a certain behavior. These formal responsibilities are orthogonal to all behavioral responsibilities.

Operation

Operation I call the responsibility of any logic. (With logic being transformations/expressions, control statements, hardware access/API calls.)

Any module containing just logic, is an operation. What an operation does, whether it stores data in a file, calculates a price, parses a string, stuffs data into a view is of no concern.

It’s implicit in that operations only contain logic. However, since this is so important, let me state it also explicitly: Operations may not call any other operation. Operations don’t know of each other. There is no functional dependency between operations. Calling one of your functions is not part of the definition of logic (see above).

Operations just execute their own logic which works on data. They are I/O-processors: given some input they produce output (and possibly side effects).

Data

Giving structure to data is a responsibility separate from operating on data. Operations of course work on data - but they are not data.

This might sound opposed to object-orientation, but it is not. Data structures may provide services to work with them. Those services just should be limited to maintaining structure and consistency of the data.

If something is data, then that’s what its functionality should be limited to. If, on the other hand, something has data, it’s free to operate on it in any way.

Integration

Since operations don’t know each other, there needs to be some other party to form a visible behavior from all those small operational behaviors. That’s what I call integration.

Integration is the responsibility to put pieces together. Nothing more, nothing less. That means, integration does not perform any logic. From this also follows, integration is not functionally dependent on operations or other integration.

Formal dependencies

The formal responsibilities (or aspects) are independent of any domain. They neither suggest nor deny there should exist adapters or controllers or use cases or business logic modules in a software. They are also independent of any requirements scenario. Software can (and should) be structured according to the formal aspects if it’s a game, a web application, a batch processor, a desktop application or a mobile application or some service running on a device.

The formal aspects are truly universal. They define fundamental formal responsibilities to be separated into modules of their own. Put operations into other modules than integration or data.

And interestingly what goes away if you do this are functional dependencies between the modules of an application.

If you decide to have some operation module for displaying data, and some for doing business calculations, and some for persisting data… then those modules won’t be calling each other.

image

They just float around independently, only maybe sharing some data.

image

As you can see, operations depend on data. Of course that’s necessary. Otherwise there would be no purpose for software. Logic needs to have raw material to process.

But data is not functionality. It’s, well, data. It’s structure.

Finally those operations floating around need to be wired-up to work in cooperation towards a common goal: some larger behavior of which each is just a part.

This is done by integration on possibly many levels.

image

Integration depends on operations and other integration. But also this dependency is not “bad”, because it’s no functional dependency.

As you can see, dependencies won’t go away. We can’t build software without them. But the functional dependencies of the behavioral architecture patterns are gone. There are no more services calling other services etc.

The dependencies here are just formal. Or empty, if you will. They don’t have to do with any functional or non-functional efficiency requirements.

Separating What from How

I call this universal separation of formal responsibilities IODA Architecture. There are several strata of Integration sitting on top of one stratum of Operations using Data - and APIs from outside the scope of the requirements to at least cause any tangible effects on hardware.

image

Or if you are more inclined towards “onions” here’s another one ;-)

image

Or this one, depending on how you want to look at it. You could say, the environment interacts with some integration, so they need to be on the surface. Or you could say, the environment is only accessible through APIs, so they need to be on the surface.

image

In any case, operations are about how behavior is created. Logic is imperative, it’s nitty gritty details.

Integration on the other hand defines what is supposed to happen. It does not contain logic, it thus is not imperative, but declarative. Like SQL is declarative. Integration assumes the stuff it integrates just to work. If that assumption is valid, then it promises to wire it up into something larger and also correctly functioning.

Also data is about the what. No logic in data except to enforce a certain structure.

Self-similarity

I’ve long sought an image to symbolize the IODA architecture. And now I’ve found one which I like quite a bit: a Sierpinski triangle.

image

I think it fits because it’s a self-similar figure. It’s a triangle made up of triangles made up of triangles… It’s triangles all the way down :-) A fractal.

IODA is also self-similar: An operation on one level of abstraction can in fact be a IODA structure - which you only see when you zoom in.

image

 

image

Operations on a high level of abstraction are black boxes. They are leaves of the the behavioral tree - which works on data and uses APIs.

But if you think, an operation is too coarse grained, you may open it up at any time. Refine it. Zoom in. Then it’s decomposed into another “miniature” IODA hierarchy. The integration formerly integrating the operation with others then integrates the root integration of the operation.[1]

Summary

To escape from “dependency hell” it’s not enough to wave the IoC/DI wand. The problem is more fundamental. It’s the very assumption software should be built as deep hierarchies of functionally dependent modules that’s problematic.

Functional dependencies lead to ever growing methods. The SRP is too weak to keep developers from adding more lines to an already excessively long method.

This is different in a IODA architecture.[2] There simply is no way to write an operational method with more a dozen lines of code. Because any developer then will tend to extract some logic into a method to be called. But that would be a violation of the IODA rule of not calling any (non-API) functions from an operation. So if a method is extracted from an operation’s logic the operation has to be turned into an integration.

Integration methods are also short. Without any logic being allowed in an integration method it simply does not feel right to have more than a dozen lines of code in it. And it’s easy to extract an integration method if the number of lines grows.

IODA does not only get rid of functional dependencies - of which a palpable effect is, the need for mock frameworks drastically diminishes -, it also creates a force keeping module sizes small.

Software architecture is not a matter of whether a Presentation Layer may only call a Business Layer or Use Cases may call Entities or the other way around. There should not be any service dependencies between these or other kinds of behavioral responsibilities in the first place.

Which behavioral responsibilities there should be is only a function of the concrete requirements. But what is truly universal and unchanging in my view is the distinction between integration, operation, data - and whatever framework APIs deemed helpful to get the job done.

image


  1. The same is true for data, by the way. Data structures are black boxes to operations. But since they can contain logic (to enforce their structure and consistency), they might not just consist of operations but a whole IODA hierarchy.

  2. Which is not just a theoretical thought, but experience from several years of designing software based on IODA.

Posted On Wednesday, April 29, 2015 3:15 PM | Comments (10) |

Monday, April 27, 2015

Sweet Aspects

When a module has a single responsibility that means it’s focused on producing the behavior of a single aspect. That’s how I described the Single Responsibility Principle (SRP) in my previous article.

Since the SRP is so important and at the same time a bit elusive, I thought, I try to illustrate this “traits and aspects thing” a bit more. Maybe you can even develop a taste for the SRP :-)

Look at this heap of sweets:

image

Let them represent logic in your code. You know, those 5,000 lines of code in a method, or those 200 methods in a class, or those 1000 classes in a library… It’s all the same. Unstructured stuff. High entropy. Disorder. Not clean. Yet at least ;-)

At some point it might have been a good thing to throw all this together. Maybe that way delivery was faster/easier? You know all those explanation how legacy code just happened over time. Should get into technical debt very consciously; have a plan how to pay it back. But in my experience that’s not how it works for teams. Like many private households (and companies) they stumble and falter and fall; or each time they they borrow it’s an exception and sure will never happen again. Promise! Until the next emergency which requires an exception.

5,000 line methods happen. They happen one line at a time. And each time there is a very good reason why to add the line instead of first refactor the whole heap of logic.

So that’s what there is at the end for 5,000 very good reasons: a huge bag of stuff. A brownfield.

image

But now is the time to bring order to the chaos ;-) Refactoring time. Let’s apply the SRP. It will make it easier to get an overview of what’s there, to make it easier to pick just the sweet you like most. Or in code: Order will make it easier to understand code, when you want to add new features or fix a bug. Also extending the code will be smoother. Less refactoring, less digging around, less unwanted side effects.

But what kind of order should there be? How to group the parts? Which traits to select to define sets with high cohesion between their elements?

I think, with regard to the depicted sweets people are divided into those who like licorice and those who don’t ;-) So “taste” could be a single trait to split the sweets into sets.

One set has the responsibility to make people happy who like licorice. The other one has the responsibility to make those happy who don’t like it. And then there is a third set with just one sweet which tastes like a gummibear, but with licorice.

image

Picking just a single trait splits a whole into as many parts as there are values the trait can assume. That might lead to a coarse grained grouping aka modularization as you can see above.

So now let’s pick two traits and see what happens, e.g. “taste” and “form”:

image

You see, this doubles the groups! There is one for (licorice, cube), (licorice, tube), (gummibear, bear), (gummibear, fruit), (gummibear, frog), (licoricegummibear, bat).

And again. Now with yet another trait: “taste”, “form”, “color”.

image

Wow! More than 20 groups of sweets. A fine grained modularization, I’d say.

The closer you look, the more traits you learn to distinguish, the smaller the groups/modules covering an aspect become. Yes, that what I’d call the selected traits: an aspect.

The combination of taste+form+color we could call the “gourmet aspect” ;-) It’s relevant for someone who relishes these kinds of sweets.

But someone who needs to pack them up (or is very hungry) might be more interested in the “size aspect” which groups according to the trait volume.

image

Or someone who wants do use sweets to lay out a picture with them might just look at the “color aspect”:

image

So as you can see, cohesion sometimes is tangible, but sometimes it’s not. For sweets color, form, size are “hard” traits. You don’t need experience with “using” the sweets in order to group them along these lines. In software it’s easily visible traits like APIs, data, logic that lead to “hard aspects”. You don’t need to understand the domain for modularization - but a scrutinizing view helps.

The trait taste is different though. You need to experience the sweets. Only then you can distinguish values and group the sweets. That’s comparable to parts of logic where just looking at it won’t help to find aspects. You have to really understand it’s parts. What’s the purpose of each line, of groups of statements? Only then you realize their “taste” and can separate them into focused modules.

The result: What formerly was just a heap of stuff - sweets or logic - now is a nicely structured, clean, modularized system of code. Each module having a clear single responsibility on one of several levels of abstraction.

image

I hope this excursion into an analogy helps you to get am even better grip on the SRP. It’s worth trying. Good modularization is at the heart of any clean coding effort.

Bon appétit! :-)

Posted On Monday, April 27, 2015 8:59 PM | Comments (0) |

Saturday, April 25, 2015

The Single Responsibility Principle under the microscope

It’s equally important and mysterious: the Single Responsibility Principle (SRP). And even though its originator Robert C. Martin tried to explain it again, I feel there is something lacking. Some concreteness, some tangibility.

First the definition of the SRP in Martin’s own words:

“The Single Responsibility Principle (SRP) states that each software module should have one and only one reason to change.”

This is more informative than Wikipedia’s tautology:

“In object-oriented programming, the single responsibility principle states that every class should have responsibility over a single part of the functionality provided by the software, and that responsibility should be entirely encapsulated by the class.”

But still… Martin’s definition is not crystal clear. It leaves much room for interpretation during TDD refactoring or code reviews. Even he admits this:

“However it begs the question: What defines a reason to change?”

To improve on this, he developed the notion of a “responsibility owner” as I’d call it:

“When you write a software module, you want to make sure that when changes are requested, those changes can only originate from a single person, or rather, a single tightly coupled group of people representing a single narrowly defined business function.”

That’s a clever idea - but it’s only useful for pretty coarse grained responsibilities. Take an online shop for example. I assume we can agree on such a shop having a couple of distinct parts like catalog, shopping cart, recommendation engine, order processing, billing etc.

image

For each of those parts there probably is a person (stakeholder) owning it, i.e. having a clear vision of how this part of the shop should behave and look like. According to Robert C. Martin this would mean, I guess, each functionality should be put into a separate module.

Who would disagree?

But that’s not really a difficult decision. Yet.

Now, what about a small company ordering such an online shop. There is only a single person responsible for all the parts. Would that mean everything has to go into just one module? Certainly not. Because what Robert C. Martin probably means it not person, but role.

Even if there is only a single responsible person for so many different parts of the software, this person views the whole from different perspectives. Mentally she switches between different roles like accounting, sales, marketing, support, even customer.

Does that help defining what a “reason to change” is? The SRP could be rephrased like this:

Separate code into modules so that each module is owned by only a single business role.

This is not bad. But still it breaks down once you zoom in on such a module. How to apply the SRP within (!) the catalog module? The whole catalog is owned by the sales role. But since it does not seem realistic to implement the whole catalog in a single function or even class, there is no guidance from the owning role as to how to form smaller modules.

I’m sorry, but even this augmented definition of the SRP does not seem to help much. My guess is, we need to look at the matter a bit more systematically.

Requirements

First let’s be clear about what responsibilities are: If a module has a single or even multiple responsibilities, then that means it contributes to the fulfillment of requirements. A module can be responsible for some functional requirement and/or some non-functional efficiency requirement (e.g. performance, scalability).

Contribution to the fulfillment of such requirements means, the logic in a module produces a part of the desired behavior. Because that’s what software is all about: behavior. Customers want software to behave in a certain way, e.g. show the products in a certain category when clicking on an image (functional requirement) or display a query result in less than 1 second (efficiency requirement).

Behavior is triggered by some input from a user. It consists of output to the user and/or side effects (e.g. changes to a database or printing a document).

imageBehavior is produced solely by logic, which to me are transformational statements (expressions), control statements (e.g. if, switch, for, repeat), and API calls/hardware access.

That’s it. That’s what customers want. Nothing more, nothing less. Which means, they don’t want modules. Not a single one.

Modules

To wrap logic up into modules does not contribute to the behavior of software. So why do it? Why should we wreck our brains about whether a module has already just a single responsibility or not?

We need modules for the non-functional requirements of evolvability and productivity. That’s their sole purpose. Modules don’t contribute to functionality or efficiency; they only make it easier to keep adding features to the code and fixing bugs.

Customers want that, too. Unfortunately they take it for granted. And we developers have a hard time to explain why keeping evolvability high is not that easy. Especially because there is no hard-and-fast metric for evolvability.[1]

So there are now orthogonal responsibilities: the responsibility with regard to functional and efficiency requirements - I call that logic responsibility -, and the responsibility with regard to evolvability.

Following the SRP thus means to identify some logic responsibility and wrap it up into a module to fulfill the evolvability responsibility.

image

Hierarchies

From the above example it should be obvious there is not just a single level of responsibilities. Responsibilities can be coarse grained or fine grained.

A module representing the catalog functionality of an online shop has a pretty broad, but still single responsibility.

But this functionality consists of different sub-functionalities and efficiencies. I imagine there to be maybe a query engine and some price calculation and a cache and several other parts to play together in order to give users a “catalog experience”. And although they are all owned by one role they should not go into the same module, I’d say.

That means a hierarchy of requirements has to be mapped onto a hierarchy of modules. But should those be all of the same kind, e.g. classes? That would dilute the hierarchy. It would not be visible in the code base anymore, at least not that easily. There would be just a bunch of classes (for query engine, calculation etc.), of which maybe one would be conceptually on a higher level (for the overall catalog).

That would work, but I suggest we broaden our view of modules. Let’s think of modules forming a hierarchy of their own.

In his first version of the SRP Robert C. Martin spoke of classes. In the updated version he speaks of modules - but leaves it to the reader to figure out what modules are. I think that doesn’t make it easier to understand what the SRP means.

So here is my take on what modules are:

Modules are design time code containers to enable evolvability and productivity.

This might sound a bit lofty, but in fact it’s pretty simple. To make it more concrete here is my hierarchy of modules from fine grained to coarse grained:

  1. Functions
  2. Classes
  3. Libraries
  4. Components
  5. (µ)Services

image

A level n module can be said to physically contain level one or more n–1 modules. Of course each level has its own characteristics. It brings something special to the table: Functions abstract logic behind a name, classes encapsulate details about data and functions, libraries are true black boxes, components separate contract from implementation, and (µ)Services sport platform neutral contracts.

Now think of it: The sole purpose of all these containers is to make it easier for your to maintain your code. They are used to encapsulate details, they save time by letting you re-use them, they decouple through contracts…

None of these containers was invented to improve functionality or performance or scalability or security.[2]

Even µServices, the newest kind of module, are not a matter of scalability or security. They are supposed to improve evolvability at design time and runtime and make polyglot development easier.[3]

With such a physical hierarchy of modules it’s easier to map a hierarchy of required behavior to code. For example the catalog could become a (µ)Service. Then the query engine and the calculation could become components within this (µ)Service. And the even smaller behavioral parts could become libraries and classes and functions. All nested within each other like russian dolls.

With a physical hierarchy of modules this can be done. It helps to map the reality of hierarchical requirements. But of course even this comes to an end. There might be more behavioral levels in the requirements than levels of modules. So in the end modules of the same level need to be arranged in some kind of logical hierarchy.

Anyway, each module on each level is supposed to have a single responsibility. But if there are maybe 5 components in a software system with 3 libraries each with 10 classes with 10 functions each… that would amount to 1,500 modules. And I don’t think you can or want to find 1,500 persons (or roles) out there who own these modules.

Such a simple calculation makes it obvious: The notion of “responsibility equals owner” is only helpful for very coarse grained modules. Don’t neglect it, but don’t put all your responsibility eggs in this basket.

Aspects

The field the SRP targets now has more structure, it has become more detailed. But my feeling is, we’re still close to square one. Robert C. Martin’s question has not been answered yet:

“What defines a reason to change?”

Which logic to put into this function and not into that one? Which functions to split among two classes instead of keeping them in just one? Which classes to group in a library? Why split a libary into three? etc. That’s the questions we have to ask during design/refactoring all the time.

Let me try to move closer to an answer by introducing the notion of an aspect. The term is not new to software development, but I want to use it in a more general way than AOP (Aspect Oriented Programming) does.

Every module has a purpose. In some way it contributes to the overall behavior of a software. The more it focuses on a certain aspect of the behavior, the better. Then it has a clear responsibility.

A module’s responsibility is to implement an aspect of requirements.

An aspect can be functional or non-functional. Aspects form a hierarchy, they can be on different levels of abstraction.

The above mentioned catalog, shopping cart etc. are aspects of the overall solution. They are easy to identify, but below that…? They are just examples. So what’s the definition of aspect? Mine is:

An aspect is a set of traits on a common level of abstraction which are likely to change together.

This might sound complicated, but in fact we deal with aspects every day. Think of a person. What are aspects of a person?

I’d say for example hairdo and clothes are aspects of a person. Each stands for a set of traits, and each trait can take on one of several values, e.g.

  • Hairdo: hair color (black, blond, red…), hair length (long, short, bald…), hair style (curly, lank, pony tail…) and more.
  • Clothes: color (black, brown, purple…), style (casual, festive, rugged…), fabric (denim, silk, linnen…) and more.

image

What binds traits together to form an aspect, what makes up their high cohesion is how they change. We view traits as cohesive when they are affected by the same causes which means they tend to change at the same time.

It’s more likely to change the hair length at the same time as the hair color and the hair style. This happens when you go to a hair dresser. The synchronicity is high, the frequency every couple of weeks. The cause might be you want to look nicer for a party or because bringing your hair in order every morning has become exhausting ;-)

It’s also more likely to change the color of your clothes at the same time as the style and the fabric. This happens when you dress yourself in the morning or dress up to go to the opera. The synchronicity is high, the frequency a couple of times per day. The cause might be you want to feel more comfortable or more appropriate for the occasion.

On the other hand it’s less likely to change your hair style and at the same time also change the color of your clothes. At least it’s that way for the people I know ;-) Which means: If there are circumstances which lead to changing hair style and clothes color more or less together, if there is a common cause for both, then… well, then it’s probably useful to view them as an aspect.[4]

image

This kind of abstraction we do every day. There are thousands of already agreed upon common aspects.

But when you are confronted with software requirements… then there are no aspects at first. It’s just a vast land of traits. It’s tons of details without abstractions. That’s why the SRP is at the core of our profession. We’re aspect hunters.

Here’s the analogy again:

First there is just a bunch of traits: hair color, clothes color, hair length, fabric etc.

Then we see those traits change their values: hair color goes from black to blond, hair length goes from long to short - but clothes color stays the same.

Then we realize there are patterns in those changes.

Then we abstract from the traits. We group them together according to the observed (or imagined) change patterns (which represent causes). Aspects are born.

In a myriad of real (or anticipated) changes to traits we need to spot patterns and then abstract. Aspect hunters are masters of abstraction. That’s what the SRP is about.[5]

Sure, ultimately all changes originate from people. That’s why Robert C. Martin says:

“[The SRP] is about people.”

But this is only helpful at the beginning. It’s a shortcut to abstraction. We don’t want to wait for changes to happen before we venture into abstraction. We want the abstractions to be served on a silver platter.

There is nothing wrong with shortcuts. But we should recognize them as such - an be prepared to go beyond them.

Hunting for aspects top-down

Spotting cohesion is no easy feat. It becomes easier with experience, though. Fortunately. But still, it’s tough. So we should look for help.

Robert C. Martin provides help when he recommends to look to stakeholders and their roles to find aspects that are worth to be represented by a module. Such roles as owners of a coarse grained aspect mostly help to structure the domain of a software into coarse grained modules.

Take the online shop for example again. There will be a person most interested in the presentation of the products, and a person responsible for the pricing, and a person responsible for order processing, and a person for billing etc. What should you make of them?

Since that’s pretty coarse grained aspects of the problem domain I find it natural to derive from this high level modules, e.g. (µ)Services or components. A catalog service or pricing component would make sense to me, but probably not a class to represent the whole aspect of price calculation.

This is an example of finding aspects top-down. You start with a business role as the root and drill down into to find requirements aspects.

When you look at patterns like the layered architecture or Ports-and-Adapters or Clean Architecture you find more examples of top-down aspect determination.

image

These patterns are congealed experience telling us: There are aspects common to all software regardless of the domain. Let’s call them cross-cutting aspects as opposed to domain aspects.

For example, interacting with the user is an aspect to be separated from persisting data or communicating with other software. These should be wrapped into modules. But of what size, on what level of the module hierarchy?

I think, such aspects should be encapsulated by at least a library; to view them as black boxes helps testing and exchanging them to keep up with technological development. These aspects form the interface of a software.

But there is more: Robert C. Martin zooms in on what’s hidden behind this interface. That’s use cases and entities to name just two aspects which are less tangible.

Whereas interface aspects can be identified pretty easily by looking at hardware or APIs, domain aspects are harder to spot. But one thing should be clear: Activities are not the same as data. That’s why use cases should be viewed as aspects of their own separted from the entities they work on.

And to represent those aspects classes seem to be the module level of choice.

How to find such “soft” aspects? Talk to the business roles, understand their domain. Certainly each role will give you ideas for lots of different such aspects. But that does not mean they all belong in just one module. Split them up!

And be open to the possibility that the role won’t be able to lead you to all of them. As your understanding of a domain evolves you will see patterns of change in traits - and will realize there are more and the existing aspects should be cut differently. That’s what Eric Evans calls “refactoring to deeper insight”.

Universal aspects

In addition to the interface and domain aspects I’d like to point out three more aspects which to me are orthogonal to the ones presented so far:

  • Integration
  • Operation
  • Data

I think they are so fundamental, they even come before the other aspects. They are the main shearing layers because they “evolve in different timescales”.[6]

image

Thinking about user interface or persistence or use cases already is pretty specific. It’s about behavioral aspects.

But behavior needs to be coordinated. And behavior needs to be fed with material to work on. To me this distinction is universal. It comes before any of the above mentioned architectural patterns.

Behavior is produced by logic. Logic is encapsulated by modules I call operations. Operations can do some business calculation, query a database, encrypt data, draw a graph or whatever.

Operation is different from data structures. Sorry, OO-folks ;-) Yes, I think for the past 25 years too much data has been entwined with too much logic. We need to start separating data from logic. Data is a very different aspect compared to behavior. Data is about structure, behavior is about process.

And then all the operations need to be “orchestrated”, put into the right order to produce an overall behavior. I call that integration. And it also is a distinct aspect different from data and operations.

Think about a company: There are documents (data) to be processed by clerks (operation) who are overseen by managers (integration). That’s even the case in software development. A product owner produces user stories (data) to be transformed into code (data) by developers (operation). And the whole process is overseen by a Scrum master (integration).

And it’s an anti-pattern if managers (or leaders) try to do the work of the employees they manage. Remember micro-management? ;-)

In code this happens if an integration module does not just “wire-up” other modules to form a process, but also contains logic to add itself to the behavior.

How to translate these universal aspects? Functions and classes are the starting point for me. Focus them on one of these aspects. Let the distinction between these aspects be the fundamental organizing principle for higher level modules/aspects.

Abstracting aspects bottom-up

So much for finding aspects top-down. But there are only so many you can identify this way. But we need to spot more, many more. Because in the end, each function we write should have just a single responsibility. Because a function is a module, the smalles module there is.

That’s where bottom-up aspect hunting starts.

It’s hard to give a definition of an aspect a priori, but it’s comparatively easy to see it as a pattern once there are traits on the table. It’s like the difference between an artist and a spectator. I find a difficult to create a good painting; whereas “good” just means “I like it” ;-) But it’s easy to recognize one when I see it.

So don’t start with the usual object oriented dance around higher level modules like classes - except for the obvious ones. Don’t play the CRC card game.

Instead start from the bottom up. Start with functions. Which means: Start with genuine behavior. All else will follow from there.

Pick some behavior requested by the customer and decompose it. What are the processing steps necessary to transform input into output plus side effects? Then repeat this for each processing step found - until the leafs of this decomposition tree become so small, their implementation is obvious to you.

imageThe processing steps you identify form flows, data flows. Integrating them into a whole is declarative programming, no logic involved. Only the leafs contain logic and only logic (operations). I call this approach “radical object-orientation” or “OOP as if you meant it”. The result of such behavior decomposition are “objects” as Alan Kay thought about them: logic encapsulated in a way so that it only communicates with the outside through messages.[7]

Anyway, what you get are many functions “without a home”. You are at the bottom of abstraction. Now build up from this higher level modules. Take each function as a trait. Which traits seem to be cohesive? Shared state or usage of the same API are signs of cohesion. Likewise focus on different requirements like functionality vs. efficiency. Or within efficiency performance vs. security vs. scalability. Or within functionality calculation vs. transformation or command vs. query. Or of course: integration vs. operation.

Slice and dice the functions (processing steps) in your decomposition hierarchy like you see fit according to observed patterns. First group them into classes. Then group classes into libraries. If you can nest classes in your language, use it to group classes within classes to form ever more coarse grained aspects. The same holds for functions.

But be careful! Within an aspect the traits should be on roughly the same level of abstraction. Abstracting aspect clothes from color and style is fine. But clothes color and collar form seem to be traits on different levels.

This is were experience is needed. Beyond hard and fast rules like suggested above there is a large area where the cohesion between traits and their level of abstraction need to be explored. Try to group them this way, then that way, and feel what makes more sense. Don’t try to avoid “refactoring to deeper insight” later on. Often you can’t be right the first time.

The one reason to change

So where are we after all of this? What about the “one and only one reason to change” which is the hallmark of a module’s responsibility?

You have to learn to spot traits in your code. Like read, write, command, query, calculate, store, data, behavior, encryption, integration, operation, caching, communication, persistence, catalog, billing, logging etc. etc. First see those traits, coarse grained and fine grained, functional as well as non-functional, domain specific vs. cross-cutting.

And then… compare them. Compare them with regard to what causes them to change. And if they have a common cause compare them with regard to how likely it is they change at the same time.

If two traits have a common cause for change and are likely to change at the same time then encapsulate them in a module on an appropriate level. You have defined an aspect and the module’s single responsibility is to implement it.

Learn to do this bottom-up. But don’t set recommendations for common aspects at naught. You find them in patterns and architectural models. Then you can hunt for aspects top-down.


  1. No, don’t even think of metrics like cyclomatic complexity or afferent coupling etc. They do not unambiguously measure evolvability. Also no customer orders evolvability in terms of such metrics. “I want functionality F, with performance P, and evolvability E.” Customers just silently assume the software they order to be easily changeable for an indefinite time. And if that’s not the case, if legacy code problems creep up and slow down development, they are always surprised. As Dag Sjøberg, a Norwegian professor in software engineering, suggests, the only metric to predict evolvability is the number of lines of code. So what customers could say is: For given functional and efficiency requirements keep the LOC at a minimum. But even that I haven’t heard of from customers. They are simply not really aware of the fundamental problem of keeping code bases evolvable.

  2. Ok, maybe with the exeption of functions. Functions where invented to save memory which might have an impact on performance, because more logic can be kept in RAM.

  3. Servers on the other hand, not (µ)Services, serve efficiency requirements like scalability or security. Servers are not modules. They are runtime containers to enable non-functional requirements and I call them hosts to distinguish them from modules. In software architecture they should not be confused with modules. They require independent design consideration.

  4. Hairdo and clothes are pretty obvious aspects, I guess. But if aspects are not hard facts, but abstractions resulting from observations and as such pretty subjective, couldn’t there be an aspect where hair style and clothes style are highly cohesive? Yes, of course. Just think of when hair style and clothes style pretty much change in sync due to a common cause. When is that the case? Say, a child becomes a teenager. Or an adult changes from hippie to conservative. Or a manager becomes a monk. How could this aspect be called when hair style and clothes style (and probably other traits as well) change in unison? How about “worldview”?

  5. If you think, aspects as abstractions are like classes, you’re right. Classes were introduced to be able to bind together what belongs together, to formally express cohesion. But classes are not the only means to define aspects. Aspects can be smaller than what should be encapsulated by a class, and they can be larger. That’s why we should understand modules to form a hierarchy like the one described above.

  6. Which is also true for many aspects, especially the coarse grained. Interface aspects evolve in a different timescale than domain aspects; the UI interface aspect evolves in a different timescale than the persistence aspect. etc.

  7. But I don’t want to delve into this right now. Let it suffice to say: This is not the functional decomposition from the last century. It’s a different kind of design approach, it’s flow design.

Posted On Saturday, April 25, 2015 5:47 PM | Comments (1) |

Wednesday, December 31, 2014

In need of more abstraction

The ultimate product of software development is this: CPU executable binary code.

image

Decades ago we used to “write” this more or less directly into memory. But that was very tedious and error prone. Code was hard to reason about, hard to change.

Abstractions in code

So we looked for ways to make coding easier. Enter a higher level of abstraction: Assembler.

image

By representing machine code instructions as text and throwing in macros productivity increased. It was easier to read programs, easier to think them up in the first place, and they were quicker to write with less errors.

According to Jack W. Reeves Assembler source code was a design of the ultimate code which got built by an automatic transformation step.

Soon, though, software was deemed hard to write even with Assembler. So many details needed to be taken care of again and again, why not hide that gain behind some abstractions?

That was when 3GL languages were invented like Fortran or Algol, later C and Pascal etc. But I want to paint this evolution differently. Because from today’s point of view the next level of abstraction on top of Assembler is not a 3GL like Java or C#, but an intermediate language like Java Byte Code (JBC) or the .NET Intermediate Language (IL).[1]

image

Solving the problem of overwhelming details of concrete hardware machines was accomplished by putting a software machine on top of it, a virtual machine (VM) with its own machine code and Assembler language.

Where Assembler provided symbolic instructions and names and macros to abstract from bits and bytes, VMs for example provided easier memory management. Not having to deal with CPU registers or memory management anymore made programming a lot easier.

Now IL Assembler source code was a design for IL byte code, which was source code for machine code Assembler, which was source code for the ultimate machine code. Ok, not really, but in principle.

IL made things simpler - but not simple enough. Programming still left much to be desired in terms of readable source code and productivity. Partly the solution to that were libraries. But also another level of abstraction was needed. Enter 3GLs with all there control structures and syntactic sugar for memory management and sub-program access.

image

That’s where we are today. Source code written in Java or C# is the design for some IL Assembler, which is the design for IL VM byte code, which is the design for machine code Assembler, which is the design for the ultimate machine code. OK, not really, but in principle.[2]

Abstraction beyond code

We like to think of 3GL source code as the design for the executable machine code. As it turned out, though, yesterday’s design, yesterday’s source code became today’s target code.

Yesterday’s abstractions became today’s details. Nobody wants to reason about software on the level of abstraction of any Assembler language. That’s why Flow Charts and Nassi-Shneiderman diagrams were invented.

And what was pseudo-code once, is now a real programming language.

Taking this evolution as a whole into view it begs the question: What’s next?

There is a pattern so far. As many levels of abstractions as have been put onto each other there is one aspect that hasn’t changed. All those languages - Assembler, IL, 3GL - are all about control flow.

Mainstream reasoning about software hasn’t changed. Today as in the 1950s it’s about algorithms. It’s about putting together logic statements to create behavior.

So how can this be extended? What’s our current “pseudo-code” about to be turned into source code of some future IDE?

My impression is: It’s over.

Control flow thinking, the imperative style of programming is at its limit.

There won’t be another level of abstraction in the same vain. I mean language-wise. The number of frameworks to be glued together to form applications will increase. There will be more levels of abstractions.

But to actually design behavior, we will need to switch to another paradigm.

Accessing data has become hugely more productive by the introduction of declarative programming languages like SQL (and modern derivatives like Linq) or Regular Expressions.

So my guess is, we need to go more in that direction. Programming has to become more declarative. We have to stave off imperative details as long as possible.

Functional Programming (FP) seems to be hinting in that direction. Recursion is a declarative solution compared to loops. Also simple data flows as f |> g in F# have declarative power because they leave open whether control flows along with data. f could (in theory) still be active while g already works on some output from f.

Still, though, even with FP there is one question unanswered: How do you think about code?

Is there a way for us to express solutions without encoding them as heaps of texts right away? Is there a way to communicate solutions without and before actually programming them? Can we describe software behavior in a systematic way on a next level of abstraction - and then systematically translate this description into Groovy or Haskell?

Object-orientation (OO) has given us more ways to describe data structures than most developers know. Think of all the relationship types defined in UML.

But software firstly is not about data structures, it’s about functionality, about behavior, about activities. How can that be described, planned, designed above today’s source code, even FP source code?

Because if Assembler, the code design of the 1950s, nowadays is just the output of a compiler translating today’s 3GL source code design… then what kind of design can be translated into today’s source code as a target?

Model-Driven Software Development (MDSD) seems to be trying to answer this question. But despite all efforts it has not been widely adopted. My guess is, that’s because the design of a modelling language is even harder than the design of a decent framework. Not many developers can do that. Also, not many domains lend themselves to this. And it’s not worth the effort in many cases.

But still, MDSD has gotten something right, I guess. Because what I’ve seen of it so far mostly is about declarative languages.

So the question seems to be: What’s a general purpose way to describe software behavior in a declarative manner?

Only by answering this question we’ll be able to enter a next level of abstraction in programming - even if that currently only means to enable more systematic designs before 3GL code and without automatic translation.

We have done that before. That’s how we started with object-orientation or querying data. First there was a model, a way of thinking, the abstraction. Then, later, there was a tool to translate abstract descriptions (designs) into machine code.

The above images all show the same code.[3] The same solution on different levels of abstraction.

However, can you imagine the solution on yet another level of abstraction above the 3GL/C# source code?

That’s what I’m talking about. Programming should not begin with source code. It should begin with thinking. Thinking in terms of models, i.e. even more abstract descriptions of solutions than source code.

As long as we’re lacking a systematic way of designing behavior before 3GL source code - be it OO or FP - we’ll be suffering from limited productivity. Like programmers suffered from limited productivity in the 1950s or 1990s before the invention of Assembler, IL, 3GLs.

And what’s the next level of abstraction?

In my view it’s data flow orientation. We have to say goodbye to control flow and embrace data flows. Control flow will always have its place. But it’s for fleshing out details of behavior. The big picture of software behavior has to be painted in a declarative manner.

Switching from OO languages to FP languages won’t help, though. Both are limited by textual representation. They are great means to encode data flows. But they are cumbersome to think in. Nobody wants to design software in machine code or byte code. Nobody wants to even do it in Assembler. And why stop with 3GLs?

No, think visually. Think in two or three or even more dimensions.

And once we’ve designed a solution in that “space”, we can translate it into lesser textual abstractions - which then will look differently.

This solution

image

surely wasn’t translated from a design on a higher level of abstraction. How the problem “wrap long lines” is approached conceptually is not readily understandable. Even if there were automatic tests to be seen they would not explain the solution. Tests just check for certain behavior.

So, as an exercise, can you imagine a solution to the problem “Word Wrap Kata” on a higher level of abstraction? Can you depict how the expected behavior could be produced? In a declarative manner?

That’s what I mean. To that level of discussion about software we have to collectively rise.

 

PS: Ok, even though I did not want to elaborate on how I think designing with data flows can work – you find more information for example in my blog series on “OOP as if you meant it” –, I guess I should at least give you a glimpse of it.

So this is a flow design for the above word wrapping problem:

 

image

This shows in a declarative manner, how I envision a process for “producing” the desired behavior. The top level/root of the hierarchical flow represents the function in question. The lower level depicts the “production process” to transform a text:

  • First split the text into words,
  • then split words longer than the max line length up into “syllables” (slices).
  • Slices then are put together to form the new lines of the given max length.
  • Finally all those lines are combined into the new text.

This sounds like control flow – but that’s only due to the simplicity of the problem. With slight changes the flow design could be made async, though. Then control would not flow along with the data anymore.

The data flow tells a story of what needs to be done, not how it exactly should happen. Refinement of a flow design stops when each leaf node seems to be easy enough to be written down in imperative source code.

Here’s a translation of the flow design into C# source code:

 

image

You see, the design is retained. The solution idea is clearly visible in code. The purpose of Wrap() is truly single: it just integrates functions into a flow. The solution can be read from top to bottom like the above bullet point list. The code is “visually honest”.

Such a flow design can be easily drawn on a flipchart. With it I can communicate my idea of how to create a certain behavior quickly to my fellow team members. It’s easy to translate into code. And since it does not contain imperative logic, it leads to very clean code, too. Logic is confined to functionally independent leafs in the decomposition tree of the flow design. Read more about this in my blog series “The Incremental Architect’s Napkin”.


  1. I even remember P-Code already used in the 1970s as the target of Pascal compilers.

  2. Of course, this is not what happens with all 3GL languages. Some are interpreted, some are compiled to real machine code. Still, this is the level of abstraction we’ve reached in general.

  3. Well, to be honest, the first image is just some arbitrary binary code. I couldn’t figure out how to get it for the Assembler code in the second image.

Posted On Wednesday, December 31, 2014 12:39 PM | Comments (1) |

Sunday, November 30, 2014

The Steep Curve of Feature Prioritization

How to prioritize features for development? Which to do first, which then, which last? That’s a much debated aspect of software development. And most teams I know are not very systematic about it.

That’s a pity, because doing features in the “wrong” order means creating value for the customer slower than possible. Or it even means producing waste.

In his recent book “The Nature of Software Development” Ron Jeffries showed how he thinks prioritization should be done with nice drawings like this:

image

Each rectangle represents a feature with a certain value to be produced with a certain effort (time, money).

image

The higher the “feature box”, the more value is produced. The longer the box, the more it costs to produce the value.

As Ron Jeffries’ drawing clearly shows, there are several ways how to order (prioritize) features. In the end, the same value is produced over the same time. But how fast how much value is added may differ greatly.

Prioritize by Value

His suggestion is: Implement high value features first, postpone low value features as long as possible. This makes for a steeper value growth curve. In the above drawing the top order of features is to be preferred. It grows value faster than the bottom order.

I agree.

But is it really that easy? Just order features according to value and effort? Take this assortment of features for example:

image

Features s..x are of the same value - but require increasing effort. Features d..a are of decreasing value - but require the same effort. The order in which to build thus is in decreasing value and growing effort.

image

I think, this is what Ron Jeffries had in mind:

Look at the difference in the growth of value if we choose the higher value inexpensive features first and defer lower value costly features until later.

But, alas, features usually don’t come in such nicely cut variations. The range of values and efforts is larger. Take the following features for example:

image

Building them in this order leads to this feature value growth curve:

image

Still not bad, isn’t it. A concave curve showing how value is produced quickly right from the start.

Prioritize by Weight

However… We can do better. And that’s what’s missing from Ron Jeffries’ book. Ordering features by value and effort easily leads to suboptimal curvature.

Let me show you what I mean. Here’s an alternative ordering for the same features:

image

Value growth looks steeper, doesn’t it?

How is this? What’s the criteria for ordering the features? It’s obviously not just value, because for example feature c is done before x which provides higher value.

The criteria is called weight. It’s calculated as described by the Weighted Shortest Job First (WSJF) method. It takes into account not only value, but also effort.

weight = value / effort

The higher the weight, the higher the feature should be prioritized.

In a diagram weight can be found in the angle of the diagonal of a feature box. Or to be precise: Since we don’t know the angle, it is the tangent of the angle.

image

The larger the angle, the steeper the diagonal points upwards, the higher the weight, the earlier the feature should be implemented.

What this means for prioritizing the above features shows the following figure where the features are ordered according to their weight:

image

You see: the angle progressively becomes smaller, the inclination of the diagonals decreases. And that means, the value growth curve is steeper, when you implement the features in this order.

Compare the value-focused prioritization drawn with triangles

image

to the WSJF prioritization:

image

The growth of value is steeper and smoother with the WSJF prioritization, there are no slumps in the curve.

Of course, value-focused prioritization and WSJF prioritization result in the same order for features of same effort. So the question is: Can you slice down requirements to be of the same effort - and still “calculate” a meaningful value for each of them?

I’d say, that’s possible - but only pretty late in the game. You already have to be very clear about a lump of requirements to break it down into (roughly) equally sized increments.

That means, mostly you’ll need to prioritize the hard way and do it in WSJF manner.

Finding Value

However, regardless how you prioritize, you need to find the value of your features. How to do that?

In my experience it’s rarely possible to find a monetary value. For large features (whole applications or modules) this might work, but not for smaller features. How many more customers will buy your software if you fix this bug or add the PDF export? How many customers will cancel the subscription to your online service, if you don’t fix the bug or improve usability?

“Cost of delay”, i.e. how much money you’ll loose/not earn as long as a feature is missing, is very hard to determine. I can’t think of any client of mine who would know (on a regular basis).

So what’s the alternative? Choose whatever fits the bill. If you’ve several thousand users you can speculate about how many of them will benefit from a feature. Or even with a few users you can ask yourself, how often they would use that feature on average (each day or each year). Or you can think about how crucial a feature is for their business. Or you can check for dependencies between features; a feature other features depend on might be worth more, because its kind of enabling. Or maybe lacking a feature poses a tangible risk, because the software would loose an important certification.

Find any number of criteria applicable to your software. And assign values to them. You can go with scales from 1 to 5 or the Fibonacci numbers. It’s less about precision as it is about comparability.

Maybe you find just three criteria. That’s better than just a gut feeling. If you range each from 0 to 5 the minimum value for a feature is 0 and the maximum value is 15.

Even with these few criteria talking about feature value becomes more differentiated and objective. Less guessing, more thinking.

Determining Effort

The same is true for determining the effort needed for a feature. We’re talking about estimation here. A sensitive topic for software developers and managers alike.

Calculating the time needed to do a feature with high reliability is near impossible. Ron Jeffries makes that very clear in his book and I can only agree.

So we should not try to predict absolute effort. Fortunately for prioritization it’s sufficient to just determine efforts for comparison. This feature will take x amount of time, that feature will take two times x, and another will take just half x.

Again a simple scale will do. Or go with the Fibonacci numbers again. Yes, it’s almost like with Story Points in Scrum. But don’t fall into the prediction trap! Don’t try to forecast how many features you’ll be able to deliver in a certain amount of time.

As soon as you start forecasting there will be people who take this for (future) reality and depend on the forecast to become true. That reduces your flexibility, that creates pressure. So by all means: Don’t forecast! Take effort figures as abstract numbers just for comparison. Assigning a 2 today will not mean the same when assigned to another feature next week.

Try to do it like this: When determining the effort for a number of features start by finding the smallest one. Assign to it 1 as the effort. Then determine the relative factors for the other ones, e.g. another feature takes twice as long (effort becomes 2), yet another feature takes much, much longer (effort becomes 10) etc.

As should be obvious: The smallest feature in today’s prioritization round can be much smaller or much larger than the smallest feature in the next round. So a 1 cannot be converted to a certain number of hours or days.

Instead of promising a result (“We will deliver features a, b, c in one week’s time.”), you should just promise a behavior (“We will deliver features a, b, c in order of their priority - which might even change over time.”).

Remember: It’s always good to promise like a pro! ;-)

Bottom line: When building software go for incremental value delivery. But value alone is not enough to prioritize. You need to take effort into account. You achieve the steepest growth in value when you prioritize based on feature weight, which essentially means you calculate a speed: “value delivery speed” equals feature value by implementation time. Bet on race horse features first, leave the nags for last.

Posted On Sunday, November 30, 2014 2:21 PM | Comments (0) |

Sunday, November 16, 2014

Feedback-Centric Development - The One Hacker Way

Erik Meijer got something right in his talk "One Hacker Way". There's a lot of bashing and ranting... but at the core there also is a precious diamond to be found. It's his admonition to be driven by feedback.

As software developer we should focus on production code - and let ourselves be guided by feedback.

How true! How simple! But contrary to the audience's belief it's no easy feat. He got much applause when he suggested, attendees who had not committed code recently should leave. People liked him extolling the virtues of "hacking", of focusing on code - instead of on fuzzy stuff like a process or even talking or thinking. No, it's the code, stupid!

Unfortunately they did not get the implications of this, I guess. And Erik Meijer did not tell them what that really, really means. So I'll try to describe how I see what truly and honestly focusing on code and feedback means.

Purpose

I'm sorry, but before I get to code, we need to lay a foundation. We need to be very clear about why we should produce code in the first place.

Code is a tool for our customers. Customers want to use software to achieve something, to reach a goal.

In order to be helpful, code needs to fulfill certain requirements. I see three basic requirements:

  • Software needs to be functional, e.g. a calculator has to provide addition and multiplication.
  • Software needs to be efficient, e.g. a calculator has to add and multiply much fast.
  • Software needs to be evolvable, e.g. a calculator needs to be adaptable to changing functional and efficiency requirements, maybe it has to also provide sine operation or has to become even faster.

Functional and efficiency requirements define the behavior of software which is produced by logic (for me that's transformational statements, control-flow-statements, and hardware access). Evolvability requires a certain structure which is spanned by modules of several sizes (for me that's function, class, library, component, micro-service).

What Erik Meijer means, when he favors hacking over some obscure agile way of development is, that software developers should produce code in order to create appropriate behavior and structure.

And what he means, when he says we should look for feedback is, that we should check whether the code written already delivers on the behavioral and structural requirements.

Being Feedback-Centric

Now for the fun part: If Erik Meijer is serious about feedback, he needs to emphasize that it has to be sought frequently. In fact the central and recurring question everything is revolving on is:

How can I get feedback as quickly as possible on my code?

That's what I call feedback-centric. Yes, we should focus on code. But code without feedback has no value. So we should seek feedback "at all costs". As soon as possible. Frequently. From the most relevant source.

Software development thus becomes a high frequency iterative activity:

  1. Code
  2. Check (gather feedback)
  3. Back to 1. for fixing any deficit or add more behavior/structure

Feedback-centric development thus is code-first development. I like. Don't you? As Erik Meijer said: Forget about test-first programming or even TDD. It's the production code which rules!

Tiny Steps - The No. 1 Implication

If you really, really buy this - it's about code and about feedback -, then you also have to buy the implication: Coding has to progress in tiny steps.

Because only tiny steps can get you frequent feedback. If you hack away for an hour or a day without feedback, then you're coding pretty much in the dark. Truely frequent feedback is not more than a couple of minutes away.

When you look at some requirements you have to ask yourself: What can I do to get feedback in the shortest possible amount of time? Can I feedback in 10 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes?

Not Only Code - The No. 2 Implication

The ultimate feedback of course is on code. So if you can feedback on some code in 1 minute go for it.
 
At least initially, though, it's even faster to get feedback without any code. Producing code and getting some stakeholder to check it for conformance to requirements often takes longer than simply asking a question.

Code should not be a question, but a statement of some understanding - even if that turns out to be wrong.

So as long as you've questions or are not very sure to understand what the requirements are... do not start hacking. Rather ask questions, e.g. by talking or by presenting some test cases you made up.

Incremental Steps - The No. 3 Implication

Being driven by code and feedback also means, you can't just programming any code. The code you want to write is, well, code you can get feedback on. That means it needs to be code some stakeholder can relate to.

Feedback-centric development thus means producing code incrementally. Code needs to make a difference, needs to produce some possibly very small additional value. And if that's indeed the case only a stakeholder can tell you.

Automatic Tests - The No. 4 Implication

Once you've identified a tiny increment you can start coding. That's just fine. No need to write a test first. What a relieve, isn't it? ;-)

Then, after maybe 3 minutes of writing production code, you run the code to give yourself a first round of feedback. Since you've asked a lot of questions you're somewhat of an authority on the requirements - but of course by no means ultimately decisive. The right to accept only lies with stakeholders.

But how do you run the code and check it for deficiencies? You can do that manually. That's just fine. But how frequent can you then check?

Not checking the behavior for correctness with automatic tests is a violation of the core principle of feedback-centric development. See, it's not just code, but also fastest feedback possible. You have to balance code production and feedback generation.

That means, you need to write automatic tests. Do it after you wrote your production code. That's fine. Since you only added a tiny increment there is not much to test. At least do it for every error you encounter. Reproduce the error with an automated test, then fix it. Rerun the test to get feedback if your fix actually fixed it.

Automatic tests have two feedback purposes:

  • whether the code you just wrote delivers on the required behavioral increment
  • whether other code still delivers on its behavioral requirements (regression tests)

TDD might seem to provide no benefit. But it should be clear now, that test-after, i.e. for feedback generation after hacking is a must. It's a sine qua non if you're serious about the One Hacker Way.

Testable Structure - No. 5 Implication

Now that automatic testing finally is inevitable even for the most eager hacker ;-) it should be obvious that not just any code structure will do. The code must be structured in a way as to be easily testable.

That even means, each increment should be testable in isolation. Otherwise the feedback would not be precise - which would be a violation of a fundamental principle we started out with.

What this leads to is... refactoring. Finally! Because in TDD refactoring is clearly optional. Yeah, it's a prescribed step after the test went green - but look at the TDD examples out there. They are a testament to how easy it is to jump this step.

No, TDD does not (!) exert any force to make a developer refactor her code. Everyone rather writes the next red test.

But if you're serious about the One Hacker Way, i.e. feedback-centric development, then you have to provide yourself with quick and precise feedback. And that (!) requires you to structure the code in a way to make it possible.

  1. Code some increment
  2. Write a test to get feedback on just that increment; if that's not possible, refactor enough to get it working

Feedback-centric development makes you the first consumer of your code. Eat your own structural dog food and see if it's palatable, i.e. if you can easily test the logic hanging in that structure.

Structural Review - No. 6 Implication

Manual or even automatic tests just provide feedback on behavior. But as stated above it's not just behavioral requirements the production needs to deliver on. Customers want us to write code in a sustainable way. Nobody knows what kind of changes come around in the next weeks, months, years. So our production code needs to be prepared; it needs to be evolvable.

Evolvability is a quality of the structure of the code. Traditionally it's produced by some kind of modularization. Recently some more principles have been added to reach this goal under the name of Clean Code.

However you call it one thing is for sure: evolvability is hard to measure. Automatic tests and the customer/user can comparatively easy give feedback on functionality and efficiency. But whether evolvability is high enough... they can't tell. Especially because evolvability cannot be stated in a requirements document.

Customers simply assume software to be infinitely malleable and live forever. Mostly, at least to my experience.

That means, tools measuring certain structural metrics cannot undoubtedly truth about the structural quality of software. At best the might hint at certain spots where it seems evolvability is lower than desired.

The ultimate feedback on evolvability comes only from... developers. If developers have a hard time to change a codebase, then it's hard to evolve. That simple.

How then can feedback from developers as authorities on evolvability be gathered frequently?

Firstly, the feedback is generated implicitly by adding the next increment. If the developer trying that find it difficult, he just has generated feedback - and can act on it. Refactoring is fixing an evolvability deficiency when it arises.

Unlike with TDD where there is no feedback on structure generated, and refactoring is recommend in a broad brush manner, in feedback-centric development refactoring always has a clear purpose. It's done when necessary to enable the next increment.

Evolvability is too important to leave it to a single developer, though. Sensitivity to structural quality is very unevenly distributed among developers for several reasons. That's why is helpful to get feedback from more than one developer as soon as possible.

Enter pair programming. During pair programming it's possible to focus on behavior and structure at the same time. Four eyes see more than two. So if you haven't been convinced of pair programming so far, but like the idea of The One Hacker Way... now is the time to start pair programming. It's a valuable technique to get more frequent feedback on code structure.

Equally valuable is of course the age old technique of doing code reviews. I don't think they should be replaced by pair programming. Code reviews go beyond the four eyes of the developers who wrote the code. More eyeballs simply can spot more structural flaws. Also a group can check if the structure matches a common understanding of how code should be modularized.

Even with pair programming and code reviews I feel there is something missing, though. They generate feedback on structure with different frequencies and from different perspectives. But the feedback of both is, hm, somewhat artificial.

Improving the structure to enable an automatic test carries a certain urgency. Refactoring is really needed to be able to continue according to the principle of frequent feedback. Pair programming and code reviews don't "force"  structural improvement in such a way.

That's why I suggest another technique I call code rotation (or maybe story rotation). Core rotation means, some requirement should not be fully implemented by a single developer or even pair. If coding an increment takes a day, for example every 90 minutes the eyeballs looking at it should be completely exchanged. Maybe developer A and B start, then C and D continue etc. Yes, A and B should be replaced by a fresh pair. There is a quick handover - and then the new pair has to get along alone with the codebase.

And that's the point: Only if the developer(s) working on a requirement change completely will the be honest feedback about the structure. If even one dev of the first pair remains for a second tour on the code the feedback is "contaminated". We're so prone to lie to ourselves when it comes to our own code... This can only be avoided by letting fresh eyeballs look at it.

Sometimes I employ this technique in trainings. I let developers start to work on an exercise - and then after a while they hand their code over to their neighbour on the right. You can imagine that this is no delight for anyone ;-) But why? It's this dissonance that needs to be removed from coding. It stems from non-obivous code structures.

Bottom line: Evolvability is of great importance. Unfortunately it's hard to measure. So we need to actually look at code and work with it to get a feeling for its quality. Therefore we need to establish a hierarchy of feedback cycles:

  1. Make every increment testable - refactor as needed. Frequency: continuously
  2. Make code understandable to your pair programming partner - refactor as needed. Frequency: minutes
  3. Make code understandable to your successor - refactor as needed. Frequency: hours
  4. Make code understandable for the whole team - refactor as needed. Frequency: day(s)

Software Design - No. 7 Implication

Switching pairs during development of a feature is a tough call. Sure you want to avoid it. But why? Too much context switch? Takes too long to find your way around the code of other devs to be able to continue their work?

Yeah, right. That's all difficult. But you can choose: experience that now - or sometime in the future. And it should be obvious that it becomes harder the longer it takes until somebody else looks at your code.

In order to make code rotation as smooth as possible another technique is needed. Frequent feedback gets another prerequisite: design.

Yes, I believe the reason for explicit design now should becomes apparent. Explicit design done by a group of devs or even the whole team helps to understand the code. Also it decreases the need for refactorings later.

Some modularization cannot be avoided to be done ad hoc during hacking. But quite some modularization can be done before even starting to code. It's a part of developing a solution. It's the "thinking before coding". And it has value because it makes it easier for developers to switch working on the codebase.

So forget about design "because that's how you do software development". Also forget about not doing design "because that's how real programmings code."

Explicit design is a means to an end. It's purpose is to develop a common understanding of easily evolvable coarse grained structures - in order to increase the frequency of feedback. Because you don't want to wait years to realize you're sitting on a monolith, if the next developer can tell you in a couple of hours he has a hard time extending what you left behind.

Continuous Deployment - No. 8 Implication

Ultimate feedback only comes from those who require your code to do their job. That means it must be very, very easy to give your code to them. The closed the code to the final usage environment the better.

That's why continuous deployment is so important for any software project. We need it to be a no brainer as much as possible to deploy code so we can ask just about anybody for feedback at any time.

Think about A/B deployment, think about deploying each increment separately, think about deploying only to a subset of customers... The more freedom and options you have, the better for gathering feedback.

In Closing

At first I did not like Erik Meijer's talk much. But once I saw through his polemic fireworks I realized how much truth can be found in what he said. Never mind his suggestion to treat developers as top athletes. Never mind him calling Jeff Sutherland a satan.

Let's stick to the title, the core of his message: We need to focus on code - because only that's delivering value. And we need integrate feedback into our work much more seriously - because only then we know if we're actually heading in the right direction with our code.

Forget about hype, buzzwords, and any elaborate belief system like "Agile" or "Scrum" etc. Yes, like the Buddhists are saying: "If you meet the Buddha on a road, kill him." We need to kill our Buddhas, the gurus, the dogmas. Let's do away with cargo cults.

Instead focus on the essential: production code. And get as much feedback as possible. Truely become a closed system on many levels of your daily practice and your organization.

PS: If you happen to recognize one of your favorite “agile practices” in my above description, congratulations. Of course there is value in some of them. We don’t need to throw the baby out with the bath water. My point, though, is to justify such practices starting just from production code and the need for feedback. Nothing more, nothing less. No buzzwords, no argumentum ad verecundiam.

Posted On Sunday, November 16, 2014 10:22 PM | Comments (1) |

Monday, October 27, 2014

Software development must deliver on budget - always

Yes, I mean it: we always need to meet the budget (be that time, money or whatever resource).1

This most likely is not your software development reality. So how come I´m demanding something so seemingly unrealistic, even preposterous?

Why?

The reason for the obligation to deliver on budget is simple: trust.

Software development is a social endeavor. It takes not only two to tango, but also at least two to develop and deliver software: a customer and a software developer.

To accomplish something in collaboration with other people requires trust. Trust is the foundation because you cannot do everything yourself. You need to let go of something and trust a collaboration partner. That´s the very reason for collaboration in the first place. If you were able to do something yourself, why get someone else on board?

So if there is a need for cooperation, then there is a need for trust. Even more so if the relationship between the cooperating parties is highly asymmetric: If you could do something yourself but delegate it to somebody else, you can at least check their work for quality. Less trust is needed.

But if you don´t have a clue how to accomplish something yourself, you can´t help but delegate execution and on top of that you´re unable to check every detail of how the result is produced. You´re pretty much limited to checks on the surface. So you need much more trust in this situation.

Software development mostly is a highly asymmetric endeavor. Customers don´t understand much of it, so they need to trust software developers.

How is trust created? Trust grows through reliability and steadiness. It´s delivering on promises - and even going beyond that. If your cooperation partner does what she promised, she´s reliable. If she does that again and again... you learn to trust her.

And if someone presents you with value you had not even expected... well, that´s even better.

Gifts build trust, keeping promises builds trust.

Delivering below budget is like a gift. Delivering on budget is keeping a promise.

That simple.

What´s wrong?

But why is it so common project reality to not deliver on budget? There is something fundamentally wrong with how the industry (or its customers) interprets the term "promise", I guess.

Promises are a form of contract. A contract requires all parties to freely agree on it, which requires all parties to understand the contractual clauses (in the same way) and their implications and ramifications. And a contract requires all parties to be able to live up to it.

Contracts are the foundation of civilization. The Romans recognized that and coined the term: pacta sunt servanda.

So what goes wrong in our industry with promises?

  1. Software development often is not free to enter a contract. Management states "You deliver such scope on such budget!" - and thinks a contract exists. But it does not. If a party got coerced then it´s not actually part of the contract.
  2. Software development often does not fully understand what´s in the contract, what it implies. Some parts of the contract are clear (mostly the budget), some are not clear, fuzzy, or outright not understandable (mostly the requirements). How can software development then honor the contract?
  3. Software development regularly overestimates its capabilities to deliver on what it perceives to be in the contract. It lacks experience, it neglects dependencies etc. Even though software development freely enters a contract or even defines the budget, it is unable to deliver on it.

No small wonder, so many contracts - large and small - are not honored. Promises are broken. Trust is not build or erodes. Overall quality drops to save at least a few contractual goals. Conflicts and pressure ensue.

How?

The only remedy to this recurring downward spiral, to this pattern is: Always deliver on budget. Always live up to what you promised.

But how?

I think, it´s very simple - which does mean it´s easy ;-)

Only make promises you can keep.

Ask yourself: Am I willing to bet 1000€ I will be able to keep this promise? If not, well, then you´re not in a position to make a promise. You doubt your own reliability.

This simple and obvious rule of course has a couple of implications:

  1. Spot and withstand any coercion. Do not enter any contract you have the faintest doubt whether you can deliver on - just because someone says so.
  2. Be humble with regard to your capabilities. Don´t overestimate what you know or are able to do.
  3. Be realistic in terms of control over your environment and your time. I strongly believe you cannot control more than maybe 1 or 2 days of your time. So you can´t promise anything beyond that.

So what can you promise? Stuff you´ve done already many times; that´s then called reproduction. You can only make promises on what you have considerable experience with and thus can deliver in "reproductive mode".

If you have experience with cooking Chicken Vindaloo you make a promise to deliver in maybe 60 minutes. But how far goes your experience in software development? Even if you´ve been doing it for 25 years your ability to reproduce is limited. This is not just because of ever changing technology, but because of ever changing customer requirements. They are hardly ever the same. As is your software.

Software is like a river: you can´t enter a river twice. Which means, it´s not the same river the next time you step into it. Something has changed, so have you.

Software development is hardly ever in reproduction mode. Mostly it´s a creative undertaking. Stuff needs to be analyzed, researched. Stuff needs to be engineered, invented.

So what can you promise to deliver on budget?

You can only promise your diligence. You can promise to work hard, to focus. But you cannot promise the implementation of a certain scope (functionality, quality) in a particular timeframe (budget). At least not beyond very small increments.

So here´s the bottom line:

  • Promise concrete results only in matters of reproduction.2
  • Since the circumstances for reproduction are rare, be conservative. Even if you are able to reproduce a result, it becomes difficult beyond 1 or 2 days - because you hardly have control over your time.
  • Without the ability to reproduce results promise only one thing: focus. Focus on a contractual topic regularly until the contract is fulfilled. Be persistent, be relentless - but don´t expect to know in advance when you´ll be done. Which means, be honest about this; don´t let yourself be lured into promises you can´t keep.

We can´t avoid to make promises. Promises are the foundation of collaboration. So we must stick to promises we actually can keep. Be extra careful what you promise. With regard to coding the time budget you agree on should be very, very short. And once you´re beyond reproduction mode, only promise diligence.

Who wants to work with unreliable people? Nobody. So we need to strive to become more reliable, more trustworthy. Working with software development should become a pleasure.


  1. An overrun of maybe 5% might be ok to classify a result as “within budget”.

  2. Yeah, I know what you´re thinking right know ;-) But there is more than that kind of reproduction…

Posted On Monday, October 27, 2014 11:27 PM | Comments (2) |

Powered by: