The Architect´s Napkin

.NET Software Architecture on the Back of a Napkin

  Home  |   Contact  |   Syndication    |   Login
  44 Posts | 0 Stories | 134 Comments | 0 Trackbacks

News

Archives

Post Categories

Saturday, March 22, 2014 #

In the beginning there was, well, chaos. Software had no particular anatomy, i.e. agreed upon fundamental structure. It consisted of several different “modules” which where dependent on each other in arbitrary ways:

image

(Please note the line end symbol I´m using to denote dependencies. You´ll see in a minute why I´m deviating from the traditional arrow.)

Then came along the multi-layer architecture. A very successful pattern to bring order into chaos. Its benefits were twofold:

  1. Multi-layer architecture separated fundamental concerns recurring in every software.
  2. Multi-layer architecture aligned dependencies clearly from top to bottom.

image

How many layers there are in a multi-layer architecture does not really matter. It´s about the Separation of Concerns (SoC) principle and disentangling dependencies.

This was better than before – but led to a strange effect: business logic was now dependent on infrastructure. Technically this was overcome sooner or later by applying the Inversion of Control (IoC) principle. That way the design time dependencies between layers where separated from the runtime dependencies.

image

This seemed to work – except now the implementation did not really mirror the design anymore. Also the layers and the very straightforward dependencies did not match a growing number of aspects anymore.

So the next evolutionary step in software anatomy moved away from layers and top-bottom thinking to rings. Robert C. Martin summed up a couple of these architectural approaches in his Clean Architecture:

image

It keeps and even details the separation of concerns, but changes the direction of the dependencies. They are pointing from technical to non-technical, from infrastructure to domain. The maxim is: don´t let domain specific code depend on technologies. This is to further the decoupling between concerns.

This leads to implementations like this:

For example, consider that the use case needs to call the presenter. However, this call must not be direct because that would violate “The Dependency Rule”: No name in an outer circle can be mentioned by an inner circle. So we have the use case call an interface (Shown here as Use Case Output Port) in the inner circle, and have the presenter in the outer circle implement it.

The same technique is used to cross all the boundaries in the architectures. We take advantage of dynamic polymorphism to create source code dependencies that oppose the flow of control so that we can conform to “The Dependency Rule” no matter what direction the flow of control is going in.

For Robert C. Martin the rings represent implementations as well as interfaces and calling an outer ring “module” implementation from an inner ring “module” implementation at runtime is ok, as long as design time dependencies of interfaces are just inward pointing.

While the Clean Architecture diagram looks easy, the actual code to me seems somewhat complicated at times.

Suggestion for a next evolutionary step

So far the evolution of software anatomy has two constants: it´s about separating concerns and aligning dependencies. Both is good in terms of decoupling and testability etc. – but my feeling is, we´re hitting a glass ceiling. What could be the next evolutionary step? Even more alignment of dependencies?

No. My suggestion is to remove dependencies from the primary picture of software anatomy altogether. Dependencies are important, we can´t get rid of them – but we should stop staring at them.

Here´s what I think is the basic anatomy of software (which I call “software cell”):

image

The arrows here do not (!) mean dependencies. They are depicting data flow. None of the “modules” (rectangles, triangles, core circle) are depending on each other to request a service. There are no client-service relationships. All “modules” are peers in that they do not (!) even know each other.

The elements of my view roughly match the Clean Architecture like this:

image

Portals and Providers form a membrane around the core. The membrane is responsible for isolating the core from an environment. Portals and providers encapsulate infrastructure technologies for communication between environment and core. The core on the other hand represents the domain of the software. It´s about use cases, if you want, and domain objects.

My focus when designing software is on functionality. So all “modules” you see are functional units. They do, process, transform, calculate, perform. They are about actions and behavior.

In my view, the primary purpose of software design is wire-up functional units in a way so a desired overall behavior (functional as well as non-functional) is achieved. In short, it´s about building “domain processes” (supported by infrastructure). That´s why I focus on data flow, not on control flow. It´s more along the lines of Functional Programming, and less like Object Oriented Programming.

Here´s how I would zoom in and depict some “domain process”:

image

Some user interacts with a portal. The portal issues a processing request. Some “chain” of functional units work on this request. They transform the request payload, maybe load some data from resources in the environment, maybe cause some side effect in some resources in the environment. And finally produce some kind of result which is presented to the user in a portal.

None of these “process steps” knows the other. They follow the Principle of Mutual Oblivion (PoMO). That makes them easy to test. That makes it easy to change the process, because any data flow can be deviated without the producer or consumer being aware of it.

In the picture of Clean Architecture Robert C. Martin seems to hint at something like this when he defines “Use Case Ports”. But it´s not explicit. That, however, I find important: make flow explicit and radically decouple responsibilities.

Two pieces are missing from this puzzle: What about the data? And what about wiring up the functional units?

Well, you got me ;-) Dependencies returning. As I said, we need them. But differently than before, I´d say.

Functional units of data flows like above surely share data which means they depend on it:

image

If data is kept simple, though, such dependencies are not very dangerous. (See how useful it is to have to symbols for relationships between functional units: one for dependencies and one for data flow.)

So far, wiring up the flows just happens. Like building dependency hierarchies at runtime just happens. Usually the code to inject instances of layer implementations at runtime is not shown. But it´s there, and a DI container knows all the interfaces and their implementations.

For the next evolutionary step of software anatomy, however, I find it important to officially introduce the authority which is responsible for such wiring up; it´s some integrator.

image

If integration is kept simple, though, such dependencies are not very dangerous. “Simple” here (as above with data) means: does not contain logic, i.e. expressions or control statements. If this Integration Operation Segregation Principle (IOSP) is followed, integration code might be difficult to test due to its dependencies – but it´s very simple to write and check during a review.

Stepping back you can see that my dependency story is different from the ones so far:

  • There are no dependencies between functional aspects. They don´t do request/response service calls on each other, but are connected by data flows.
  • There are only dependencies between fundamental organizational concerns completely orthogonal to any domain: integration, operation, and data.

image

This evolved anatomy of software does not get rid of dependencies. You will continue to use your IoC and DI containers ;-) But it will make testing of “work horse code” (operations) easier, much easier. And the need for using mock frameworks will decrease. At least that´s my experience of some five years designing software like this.

Also, as you´ll find if you try this out, specifications of classes will change. Even with IoC a class will be defined by 1+n interfaces: the interface it implements plus all the interfaces of “service classes” it uses.

But with software cells and flows the class specifications consist only of 1 interface: the interface the class implements. That´s it. Well, at least that´s true for the operation classes which follow the PoMO. That´s useful because those classes are heavy with logic, so you want to make it as simple as possible to specify and test them.

Conclusion

The evolution of a basic software anatomy has come far – but there is still room for improvement. As long as everything revolves around dependencies between technological and domain aspects, there is unholy coupling. So the next consequent move is to get rid of those dependencies – and relegate them to the realm of organizational concerns. In my opinion the overall structure becomes much easier. Decluttered. More decoupled.

Why not give it a chance?

 

PS: For more details on flows, PoMO, and IOSP see my blog series here – or make yourself comfortable in a chair next to your fireplace and read my Leanpub eBook on it :-)


Sunday, March 16, 2014 #

Doing explicit software architecture is nothing for the faint of heart, it seems. Software architecture carries the nimbus of heavy weight, of something better left to specialists – or to eschew altogether. Both views are highly counterproductive, though. Leaving architecture to specialists builds walls where there should be none; it hampers feedback and flexibility. And not explicitly architecting (or just designing) software leads to less than desirable evolvability as well as collective software ownership.

So the question is, how to do “just enough” software design up-front. What´s the right amount? What´s an easy way to do it?

Since 2005 I´m on a quest to find answers to these questions. And I´m glad I have found some – at least for me ;-) I´ve lost my “fear of the blank flipchart” when confronted with some requirements document. No longer do I hesitate to start designing software. I´ve shrugged off UML shackles, I´ve gotten off the misleading path of object oriented dogma.

This is not to say, there is no value in some UML diagrams or features of object oriented technologies. Of course there is – as long as it helps ;-)

But as with many practices one never reaches the finishing line when it comes to software architecture. Although I feel comfortable attacking just about any requirements challenge, it´s one think to feel confident – and an altogether different to actually live up to the challenge. So I´m on a constant lookout for exercises in software architecture to further hone my skills. That means applying my method – which is a sampling of many approaches with some added idiosyncrasies – plus reflecting on the process and the outcome.

At the Coding Dojo of the Clean Code Developer School I´ve compiled more than 50 such exercises of different sizes (from small code/function katas to architecture katas). If you like, try them out yourself or with your team (some German language skills required, though ;-).

And recently I stumbled across another fine architecture kata. It´s from Simon Brown whose book “Software Architecture for Developers” I read. First the exercise was only in the book, but then he made it public. I included it in the Coding Dojo and added line numbers and page numbers. Find the full text for the requirements of a Financial Risk System here.

image

Since Simon included pictures of architectural designs for this exercise from students of his trainings, I thought, maybe I should view that as a challenge and try it myself. If I´m confident my approach to software architecture is helpful, too, then why not expose myself with it. Maybe there are interesting similarities to be discovered – maybe there are differences that could be discussed.

Following I´ll approach the architecture for the Financial Risk System (FRS) my way. This will show how I approach a design task, but it might lack some explanation as to the principles underlying this approach. Here´s not enough room, though, to layout my whole thought framework. But I´m working on a book… ;-) It´s called “The Architect´s Napkin – The Cheat Sheet”. But beware, for the first version it´s in German.

Why it´s called the Architect´s Napkin you´ll see in a minute – or read here. Just let me say: If software architecture is to become a disciplin for every developer it needs to be lightweight. And design drawing should fit on an ordinary napkin. All else will tend to be too complicated and hard to communicate.

And now for some design practice. Here´s my toolbox for software architects:

image

Basic duality: system vs environment

Every software project should start by focusing on what´s its job, and what not. It´s job is to build a software system. That´s what has to be at the center of everything. By putting something at the center, though, other stuff is not at the center. That´s the environment of what´s at the center. In the beginning (of a software project) thus there is duality: a system to build vs the environment (or context) of the system.

And further the system to build is not created equal in all parts. Within the system there is a core to be distinguished from the rest. At the core of the system is the domain. That´s the most important part of any software system. That´s what we need to focus on most. It´s for this core that a customer wants the system in the first place.

image

That´s a simple diagram. But it´s an important one as you´ll see. Best of all: you can draw it right away when your boss wants you to start a new software project ;-) It even looks the same for all software systems. Sure, it´s very, very abstract. But that helps as long as you don´t have a clue what the requirements are about.

Spotting actors

With the system put into the focus of my attention I go through the requirements and try to spot whose actually going to use it. Who are the actors, who is actively influencing the system? I´m not looking for individual persons, but roles. And these roles might be played by non-human actors, i.e. other software systems.

  • The first actor I encounter is such a non-human actor: the risk calculation scheduler. It requests the software system to run and produce a risk report. Line 8 and 9 in the requirements document allude to this actor.
  • Then on page 2, line 54f the second actor is described: risk report consumer. It´s the business users who want to read the reports.
  • Line 56f on the same page reveal a third actor: the calculation configurator. This might be the same business user reading a report, but it´s a different role he´s playing when he changes calculation parameters.
  • Finally lines 111ff allude to a fourth actor: the monitoring scheduler. It starts the monitoring which observes the risk calculation.

The risk calculation scheduler and the monitoring scheduler are special in so far as they are non-human actors. They represent some piece of software which initiates some behavior in the software system.

Here´s the result of my first pass through the requirements:

image

Now I know who´s putting demands on the software system. All functionality is there to serve some actor. Actors need the software system; they trigger it in order to produce results [1].

Compiling resources

During my second pass through the requirements I focus on what the software system needs. Almost all software depends on resources in the environment to do its job. That might be a database or just the file system. Or it might be some server, a printer, a webcam, or some other hardware. Here are the resources I found:

  • Page 1, line 10f: the existing Trade Data System (TDS)
  • Page 1, line 11: the existing Reference Data System (RDS)
  • Page 2, line 52f: the risk report (RR)
  • Page 2, line 54f: some means to distribute the RR to the risk report consumers (risk report distribution, RRD)
  • Page 2, line 54f: a business user directory (BUD)
  • Page 2, line 56f: external parameters (EP) for the risk calculation
  • Page 3, line 92ff: an audit log (AL)
  • Page 4, line 111ff: SNMP
  • Page 4, line 117f: an archive (Arc)

image

Nine resources to support the whole software system. And all of them require some kind of special technology (library, framework, API) to use them.

The diagram as a checklist

What I´ve done so far was just analysis, not design. I just harvested two kind of environmental aspects: actors/roles and resources. But they are important for three reasons. Firstly they help structuring the software system as you´ll see later. Secondly they guide further analysis of the requirements. And last but not least they function as a checklist for the architect.

Each environmental element poses questions, some specific to its kind, some general. And by observing how easy or difficult it is to answer them, I get a feeling for the risk associated with them.

My third pass through the requirements is not along the document, but around the circle of environmental aspects identified. As I focus on each, I try to understand it better. Here are some questions I´d feel prompted by the “visual checklist” to ask:

  • Actor “Risk calculation scheduler”: How should the risk calculation be started automatically each day? What´s the operating system it´s running on anyway? Windows offers a task scheduler, on Linux there is Crontab. But there are certainly more options. Some more research is needed – and talking to the customer.
  • Actor “Risk report consumer”: How should consumers view reports? They need to be able to use Excel, but is that all? Maybe Excel is just a power tool for report analysis, and a quick overview should be gained more easily? Maybe it´s sufficient to send them the reports via email. Or maybe just a notification of a new report should be sent via email, and the report itself can be opened with Excel from a file share? I need to talk to the customer.
  • Actor “Calculation configurator”: What kind of UI should the configurator be using? Is a text editor enough to access a text file containing the parameters – protected by operating system access permissions? Or is a dedicated GUI editor needed? I need to talk to the customer.
  • Actor “Monitoring scheduler”: The monitoring aspect to me is pretty vague. I don´t really have an idea yet how to do it. I feel this to be an area of quite some risk. Maybe monitoring should be done by some permanently running windows service/daemon which checks for new reports every day and can be pinged with a heartbeat by the risk calculation? Some research required here, I guess. Plus talking to the customer about how important monitoring is compared to other aspects.
  • Resource “TDS”:
    • What´s the data format? XML
    • What´s the data structure? Simple table, see page 1, section 1.1 for details
    • What´s the data volume? In the range of 25000 records per day within the next 5 years (see page 3, section 3.b)
    • What´s the quality of the data, what´s the reliability of data delivery? No information found in the requirements.
    • How to access the data? It´s delivered each day as a file which can be read by some XML API. Sounds easy.
    • Available at 17:00h (GMT-4 (daylight savings time) or GMT-5 (winter time)) each day.
  • Resource “RDS”:
    • What´s the data format? XML
    • What´s the data structure? Not specified; need to talk to the customer.
    • What´s the data volume? Some 20000 records per day (see page 3, section 3b)
    • What´s the quality of the data, what´s the reliability of data delivery? No information found in the requirements.
    • How to access the data? It´s delivered each day as a file which can be read by some XML API. Sounds easy – but record structure needs to be clarified.
  • Resource “Risk report”:
    • What´s the data format? It needs to be Excel compatible, that could mean CSV is ok. At least it would be easy to produce. Need to ask the customer. If more than CSV is needed, e.g. Excel XML or worse, then research time has to be alotted, because I´m not familiar with appropriate APIs.
    • What´s the data structure? No information have been given. Need to talk to the customer.
    • What´s the data volumne? I presume it depends on the number of TDS records, so we´re talking about some 25000 records of unknown size. Need to talk to the customer.
    • How to deliver the data? As already said above, I´m not sure if the risk report should be stored just as a data file or be sent to the consumers via email. For now I´ll go with sending it via email. That should deliver on the availability requirement (page 3, section 3.c) as well as on the security requirements (page 3, section e, line 82f, 84f, 89f).
    • Needs to be ready at 09:00h (GMT+8) the day following TDS file production.
  • Resource “Risk report distribution”: This could be done with SMTP. The requirements don´t state, how the risk reports should be accessed. But I guess I need to clarify this with the customer.
  • Resource “Business user directory”: This could be an LDAP server or some RDBMS or whatever. The only thing I´d like to assume is, the BUD contains all business users who should receive the risk reports as well as the ones who have permission to change the configuration parameters. I would like to run a query on the BUD to retrieve the former, and use it for authentication and authorization for the latter. Need to talk to the customer for more details.
  • Resource “External parameters”: No details on the parameters are given. But I assume it´s not much data. The simplest thing probably would be to store them in a text file (XML, Json…). That could be protected by encryption and/or file system permissions, so only the configurator role can access it. Need to talk to the customer if that´d ok.
  • Resource “Audit log”:
    • Can the AL be used for what should be logged according to page 3, section 3.f and 3.g?
    • Is there logging infrastructure already in place which could be used for the AL?
    • What´s the access constraints for the AL? Does it need special protection?
  • Resource “SNMP”: I don´t have any experience with SNMP. But it sounds like SNMP traps can be sent via an API even from C# (which I´m most familiar with). Need to do some research here. The most important question is, how to detect the need to send a trap (see above the monitoring scheduler).
  • Resource “Archive”:
    • Is there any Arc infrastructure already in place?
    • What are the access constraints for the Arc? Does it need special protection?
    • My current idea would be to store the TDS file and the RDS file together with the resulting risk report file in a zip file and put that on some file server (or upload it into a database server). But I guess I need to talk to the customer about this.

In addition to the environmental aspects there is the domain to ask questions about, too:

  • How are risks calculated anyway? The requirements don´t say anything about that. Need to talk to the customer, because that´s an important driver for the functional design.
  • How long will it take to calculate risks? No information on that, too, in the requirements document; is it more like some msec for each risk or like several seconds or even minutes? Need to talk to the customer, because that´s an important driver for the architectural design (which is concerned with qualities/non-functional requirements).
  • When the TDS file is produced at 17:00h NYC time (GMT-5) on some day n it´s 22:00h GMT of the same day, and 06:00h on day n+1 in Singapore (GMT+8). This gives the risk calculation some 3 hours to finish. Can this be done with a single process? That needs to be researched. The goal is, of course, to keep the overall solution as simple as possible.

So much for a first run through the visual checklist the system-environment diagram provides.

2014-03-16 10.56.14

The purpose of this was to further understand the requirements – and identify areas of uncertainty. This way I got a feeling for the risks lurking in the requirements, e.g.

  • The larges risk to me currently is with the domain: I don´t know how the risk calculation is done, which has a huge impact on the architecture of the core.
  • Then there is quite some risk in conjunction with infrastructure. What kind of infrastructure is available? What are the degrees of freedom in choosing new infrastructure? How rigid are security requirements?
  • Finally there is some risk in technologies I don´t have any experience with.

Here´s a color map of the risk areas identified:

2014-03-16 11.17.23

With such a map in my hand, I´d like to talk to the customer. It would give us a guideline in our discussion. And it´s a list of topics for further research. Which means it´s kind of a backlog for things to do and get feedback on.

But alas, the customer is not available. Sounds familiar? ;-) So what can I do? Probably the wisest thing to do would be to stop further design and wait for the customer. But this would spoil the exercise :-) So I´ll continue to design, tentatively. And hopefully this does not turn out to be waste in the end.

Refining to applications – Design for agility

The FRS is too big to be implemented or even further discussed and designed as a whole. It needs to be chopped up :-) I call that slicing in contrast to the usual layering. At this point I´m not interested in more technical details which layers represent. I´d like to view the system through the eyes of the customer/users. So the next step for me is to find increments that make sense to the customer and can be focused on in turn.

For this slicing I let myself be guided by the actors of the system-environment-diagram. I´d like to slice the system in a way so that each actor gets its own entry point into it. I call that application (or app for short).

2014-03-16 14.41.57

Each app is a smaller software system by itself. That´s why I use the same symbol for them like for the whole software system. Together the apps make up the complete FRS. But each app can be delivered separately and provides some value to the customer. Or I work on some app for a while without finishing it, then switch to another to move it forward, then switch to yet another etc. Round and round it can go ;-) Always listening to what the customer finds most important at the moment – or where I think I need feedback most.

As you can see, each app serves a single actor. That means, each app can and should be implemented in a way to serve this particular actor best. No need to use the same platform or UI technologies for all apps.

Also the diagram shows how I think the apps share resources:

  • The Risk Report Calculation app needs to read config data from EP and produces a report RR to be sent to business users listed in BUD. Progress or hindrances are logged to AL.
  • The Config Editor also needs to access BUD to check, who´s authorized to edit the data in EP. Certain events are logged to AL.
  • The Report Reader just needs to access the report file. Authorization is implicit: since the business user is allowed to access the RR folder on some file share, he can load the report file with Excel. But if need be, the Report Reader could be more sophisticated and require the business user to authenticate himself. Then the report reader would also need access to BUD.
  • The Monitor app checks the folder of the report files each day, if a file has arrived. In addition the Monitor app could be a resource to the Report Calculation which can send it heart beat to signal it´s doing well.

The other resources are used just by the Report Calculation.

Now that I have sliced up the whole software system into applications, I can focus on them in turn. What´s the most important one? Where should I zoom in?

Hosting applications – Design for quality

Zooming in on those applications can mean two things: I could try to slice them up further. That would mean I pick an application and identify its dialogs and then then interactions within each dialog. That way I´d reach the function level of a software system, where each function represents an increment. Such slicing would be further structuring the software system from the point of view of the domain. It would be agile design, since the resulting structure would match the view of the customer. Applications, dialogs, and interactions are of concern to him.

Or I could zoom in from a technical angle. I´d leave the agile domain dimension of design which focuses on functionality. But then which dimension should I choose? There are two technical dimensions, in my view. One is concerned with non-functional requirements or qualities (e.g. performance, scalability, security, robustness); I call it the host dimension. Its elements describe the runtime structure of software. The other is concerned with evolvability and production efficiency (jointly called the “security of investment” aspect of requirements); I call it the container dimension. Its elements describe the design time structure of software.

So, which dimension to choose? I opt for the host dimension. And I focus on the Risk Report Calculation application. That´s the most important app of the software system, I guess.

Whereas the domain dimension of my software architecture approach decomposes a software system into ever smaller slices called applications, dialogs, interactions, the host dimension provides decomposition levels like device, operation system process, or thread.

Focusing on one app the questions thus are: How many devices are needed to run the app so it fulfills the non-functional requirements? How many processes, how many threads?

What are the non-functional requirements determining the number of devices for the calculation app of the FRS? It needs to run in the background (page 1, line 7f), it needs to generate the report within 3 hours (line 28 + line 63), it needs to be able to log certain events (page 3, line 94) and be monitored (page 4, lines 111ff).

How many devices need to be involved to run the calculation strongly depends on how long the calculations take. If they are not too complicated, then a single server will do.

And how many processes should make up the calculation on this server? Reading section 2, lines 43ff I think a single process will be sufficient. It can be started automatically by the operating system, it is fast enough to do the import, calculation, notification within 3 hours. It can have access to the AL resource and can be monitored (one way or the other).

At least that´s what I want to assume lacking further information as noted above.

2014-03-16 16.56.33

Of course this host diagram again is a checklist for me. For each new element – device, process – I should ask appropriate questions, e.g.

  • Application server device:
    • Which operating system?
    • How much memory?
    • Device name, IP address?
    • How can an app be deployed to it?
  • Application process:
    • How can it be started automatically?
    • Which runtime environment to use?
    • What permissions are needed to access the resources?
    • Should the application code own the process or should it run inside an application server?

The device host diagram and the process host diagram look pretty much the same. That´s because both container only a single element. In other cases, though, a device is decomposed into several processes. Or there are several devices each with more than one process.

Also this is only the processes which seem necessary to fulfill quality requirements. More might be added to improve evolvability.

Nevertheless drawing those diagrams is important. Each host level (device, process – plus three more) should be checked. Each can help to fulfill certain non-functional requirements, for example: devices are about scalability and security, processes are about robustness and performance and security, threads are about hiding or lowering latency.

Separating containers – Design for security of investment

Domain slices are driven directly by the functional requirements. Hosts are driven by quality concerns. But what drives elements like classes or libraries? They belong to the container dimension of my design framework. And they can only partly be derived directly from requirements. I don´t believe in starting software design with classes. They don´t provide any value to the user. Rather I start with functions (see below) – and then find abstractions on higher container levels for them.

Nevertheless the system-environment-diagram already hints at some containers. On what level of the container hierarchy they should reside, is an altogether different question. But at least separate classes (in OO languages) should be defined to represent them. Mostly also separate libraries are warranted for even more decoupling.

Here´s the simple rule for the minimum number of containers in any design:

  • communication with each actor is encapsulated in a container
  • communication with each resource is encapsulated in a container
  • the domain of course needs its own container – at least one, probably more
  • a dedicated container to integrate all functionality; usually I don´t draw this one; it´s implicit and always present, but if you like, take the large circle of the system as its representation

2014-03-16 17.23.22

Each container has its own symbol: the actor facing container is called a portal and drawn as a rectangle, the resource facing containers are called providers and drawn as triangles, and the domain is represented as a circle.

That way I know the calculation app will consist of 9+1+1+1=12 containers. It´s a simple and mechanical separation of concerns. And it serves the evolvability as well as production efficiency.

By encapsulating each actor/resource communication in its own container, it can more easily replaced, tested, and implemented in parallel. Also this decouples the domain containers from the environment.

Interestingly, though, there is no dependency between these containers! At least in my world ;-) None knows of the others. Not even the domain knows about them. This makes my approach different from the onion architecture or clean architecture, I guess. They prescribe certain dependency orientations. But I don´t. There are simply no dependencies :-) Unfortunately I can´t elaborate on this further right here. Maybe some other time… For you to remember this strange approach here is the dependency diagram for the containers:

2014-03-16 17.25.18

No dependencies between the “workhorses”, but the integration knows them all. But such high efferent coupling is not dangerous. The integration does not contain any logic; it does not depend itself on the operations of the other containers. Integration is a special responsibility of its own.

Although I know a minimal set of containers, I don´t know much about their contracts. Each is encapsulating some API through which it communicates with a resource/actor. But how the service of the containers is offered to its environment is not clear. I could speculate about it, but most likely that would violate the YAGNI principle.

There is a way, however, to learn more about those contracts – and maybe find more containers…

Zooming in – Design for functionality

So far I´ve identified quite some structural elements. But how does this all work together? Functionality is the first requirement that needs to be fulfilled – although it´s not the most important one [2].

I switch dimensions once again in order to answer this question. Now it´s the flow dimension I´m focusing on. How does data flow through the software system and get processed?

On page 2, section 2 the requirements document gives some hints. This I would translate into a flow design like so. It´s a diagram of what´s going on in the calculation app process [3]:

2014-03-16 18.20.43

Each shape is a functional unit which does something [4]. The rectangle at the top left is the starting point. It represents the portal. That´s where the actor “pours in” its request. The portal transforms it into something understandable within the system. The request flows to the Import which produces the data Calculate then transforms into a risk report.

That´s the overall data flow. But it´s too coarse grained to be implemented. So I refine it:

  • Zooming into Import reveals to separate import steps – which could be run in parallel – plus an Join producing the final output.
  • Zooming into Calculate releals several processing steps. First the input from the Import is transformed into risk data. Then the risk data is transformed into the actual risk report, of which then the business users are informed. Finally the TDS/RDS input data (as well as the risk report) gets archived.

The small triangles hint at some resource access within a processing step. Whether the functional unit itself would do that or if it should be further refined, I won´t ponder here. I just wanted to quickly show this final dimension of my design approach.

For Import TDS and Import RDS I guess I could derive some details about the respective container contracts. Both seem to need just one function, e.g. TDSRecord[] Import(string tdsFilename).

The other functional units hint at some more containers to consider. Report generation (as opposed to report storage) looks like a different responsibility than calculating risks, for example. Also Import and Calculate have a special responsibility: integration. They see to that the functional units form an appropriate flow.

At least the domain thus is decomposed into at least three containers:

  • integration
  • calculation
  • report generation

Each responsibility warrants its own class, I´d say. That makes it 12-1+3=14 containers for the calculation application.

Do you see how those containers are a matter of abstraction? I did not start out with them; rather I discovered them by analyzing the functional structure, the processing flow.

Retrospective

So much for my architectural design monologue ;-) Because a monologue it had to be since the customer was not at hand to answer my many questions. Nevertheless I hope you got an impression of my approach to software design. The steps would not have been different if a customer had be available. However the resulting structure might look different.

Result #1: Make sure the customer is close by for questions when you start your design.

The exercise topic itself I found not particularly challenging. The interesting part was missing ;-) No information on what “calculating risks” means. But what became clear to me once more was:

Result #2: Infrastructure is nasty

There are so many risks lurking in infrastructure technologies and security constraints and deployment and monitoring. Therefore it´s even more important to isolate those aspects in the design. That´s what portals and providers are for.

To write up all this took me a couple of hours. But the design itself maybe was only half an hour of effort. So I would not call it “big design up-front” :-)

Nonetheless I find it very informative. I would not start coding with less. Now I talk about focus and priorities with the customer. Now I can split up work between developers. (Ok, some more discussion and design would be needed to make the contracts of the containers more clear.)

And what about the data model? What about the domain model? You might be missing the obligatory class diagram linking data heavy aggregates and entities together.

Well, I don´t see much value in that in this case – at least from the information given in the requirements. The domain consists of importing data, calculating risks and generating a report. That´s the core of the software system. And that´s represented in all diagrams: the host diagram shows a process which does exactly this, the container diagram shows a domain responsible for this, and the flow diagram shows how several domain containers play together to form this core.

Result #3: A domain model is just a tool, not a goal.

All in all this exercise went well, I´d say. Not only used I flows to design part of the system, I also felt my work flowing nicely. No moment of uncertainty how to move on.

Endnotes

[1] Arguably the non-human actors in this scenario don´t really need the software system. But as you´ll see it helps to put them into the picture as agents to cause reactions of the software system.

[2] The most important requirements are the primary qualities. It´s for them that software gets commissioned. Most often that´s performance, scalability, and usability. Some operations should be executed faster, with more load, and be easier to use through software, than without. But of course, before some operation can become faster it needs to be present first.

[3] If this reminds you of UML activity diagrams, that´s ok ;-)

[4] Please not my use of symbols for relationships. I used lines with dots at the end to denote dependencies. In UML arrows are used for that purpose. However, arrows I reserve for data flow. That´s what they are best at: showing from where to where data flows.


Thursday, March 13, 2014 #

Antifragility has attracted some attention lately. I, too, pressed the “I like” button. :-) A cool concept, a term that was missing. Just when Agility starts to become lame, Antifragility takes over the torch to carry on the light of change… ;-)

What I find sad, though, is that the discussion seems to be too technical too soon. There´s the twitter hash tag #antifragilesoftware for example. It´s suggesting, there are tools and technologies and methods, to make software antifragile. But that´s impossible, I´d say. And so the hash tag is misleading and probably doing more harm than good to the generally valuable concept of Antifragility.

Yes, I´m guilty of using the hash tag, too. At first I hadn´t thought about Antifragility enough and was just happy, someone had brought up the term. Later I used it to not alienate others with yet another hash tag; I succumbed to the established pattern.

But here I am, trying to at least once say, why I think, the hash tag is wrong – even though it´s well meaning.

Antifragility as a property

Antifragility is a property. Something can be fast, shiny, edible – or antifragile. If it´s antifragile it thrives on change, it becomes better due to stress. What is antifragile “needs to be shaken and tossed” to improve. “Handle with care” is poison to whatever is antifragile.

As I explained in a previous article, I think Antifragility comes from buffers – which get adapted dynamically. Increase of buffer capacity in response to stress is what distinguishes Antifragility from robustness.

In order to determine if something is antifragile, we need to check its buffers. Are there buffers for expectable stresses? How large is their capacity before stress? Is stress on a buffer-challenging level? How large are the buffer capacities after such stress? If the buffer capacities is larger after challenging stress – of course given a reasonable recuperation period –, then I guess we can put the label “Antifragile” on the observed.

Antifragility is DIY

So far it seems, Antifragility is a property like any other. Maybe it´s new, maybe it´s not easy to achieve, but in the end, well, just another property we can build into our products. So why not build antifragile blenders, cars, and software?

Here´s where I disagree with the suggestion of #antifragilesoftware. I don´t think we can build Antifragility into anything material. And I include software in this, even though you might say it´s not material, but immaterial, virtual. Because what software shares with material things is: it´s dead. It´s not alive. It does not do anything out of its own volition.

But that´s what is at work in Antifragility: volition. Or “the will to live” to get Schopenhauer into the picture after I called Nietzsche the godfather of Antifragility ;-)

Antifragility is only a property of living beings, maybe it´s even at the core of the definition of what life is.

Take a glass, put it in a box, skake the box-with-glass, the glass cracks. The stress on the buffers of the box-with-glass was challenging. Now put some wood wool around the glass in the box. Shake the box-with-glass, the glass is not further damaged. Great! The box-with-glass is antifragile, right?

No, of course not. The box-with-glass is just robust. Because the box-with-glass did not change itself. It was build with a certain buffer. That buffer was challenged by stress. And that´s it. The box-with-glass did not react to this challenge – except it suffered it. It even deteriorated (the glass cracked).

It was you as the builder and observer who increased some buffer of the box-with-glass after the stress. You just improved its robustness.

If the box-with-glass was antifragile it would have reacted itself to the stress. It would have increased its buffer itself. That way the box-with-glass would have shown it learned from the stress.

Antifragility thus essentially is a Do-it-yourself property. Something is only antifragile if it itself reacts to stresses by increasing some buffer. Antifragility thus is a property of autopoietic systems only. Or even: Autopoiesis is defined by Antifragility.

To put it short: If you´re alive and want to stay alive you better be Antifragile.

Antifragility is needed to survive in a changing environment.

So if you want to build antifragility into something, well, then you have to bring it to life. You have to be a Frankenstein of some sorts ;-) Because after you built it, you need to leave it alone – and it needs to stay alive on its own.

You see, that´s why I don´t believe in #antifragilesoftware. Software is not alive. It won´t change by itself. We as its builders and observers change it when we deem it necessary.

Software itself can only be robust. We build buffers of certain sizes into it. But those buffers will never change out of the software´s own volition. At least not before the awakening of Skynet :-)

So forget SOLID, forget µServices, forget messaging, RabbitMQ or whatever your pet principles and technologies might be. Regardless of how much you apply them to software, the software itself will not become antifragile. Not even if you purposely (or accidentally) build a god class :-)

Enter the human

Since we cannot (yet) build anything material that´s alive, we need to incorporate something that´s alive already, if we want to achieve Antifragility. Enter the human.

If we include one or more humans into the picture, we actually can build living systems, ie. systems with the Antifragility property. Before were just technical systems, now it´s social systems.

Not every social system is alive of course. Take the people sitting on a bus. They form a social system, but I would have a hard time to call it autopoietic. There´s nothing holding those people together except a short term common destination (a bus stop). And even this destination does not cause the group of people to try to keep itself together in case the bus breaks down. Then they scatter and each passenger tries to reach his/her destination by some other means.

But if you put together people in order to achieve some goal, then the social system has a purpose – and it will try “to stay alive” until the goal is reached. Of course for this they need to be bestowed with sufficient autonomy.

I think a proper term for such a purpose oriented social system would be organization. A company is an organization, a team is an organization.

An organization like any other system has buffers. It can withstand certain stresses. But above that it is capable of extending its buffers, if it feels that would help its survival towards fulfilling its purpose. That´s what autonomy is for – at least in part.

“Dead” systems, i.e. purely material systems (including software) usually deteriorate under stress – some faster, some slower. At best they might be able to repair themselves: If a buffer got damaged they can bring it back to its original capacity. But then they don´t decide to do this; that´s what “just happens”. That´s how they are built, that´s an automatic response to damage, it´s a reflex.

“Dead” systems however don´t deliberate over buffer extension. They don´t learn. They don´t anticipate, assess risks, and as a result direct effort this way or that to increase buffer capacity to counter future stresses.

Let´s be realistic: that´s the material systems (including software) we´re building. Maybe in some lab mad scientists are doing better than that ;-), but I´d say that´s of no relevance to the current Antifragility discussion.

Software as a tool

If we want to build antifragile systems we´re stuck with humans. Antifragility “build to order” requires a social system. So how does software fit into the picture?

What´s the purpose of a software development team? It´s to help the customer respectively the users to satisfy some needs easier than without it. The customer for example says: “I want to get rich quick by offering an auction service to millions of people.” The customer then would be happy to accept any (!) solution, be that software or a bunch of people writing letters or trained ants. As it happens, though, software (plus hardware, of course) seems make it easier than training ants to reach this goal :-) That´s the only reason the “solution team” will deliver software to the customer. Software thus is just a tool. It´s not a purpose, it´s not a goal, it´s a tool fitting a job at a specific moment.

Of course, a software team would have a hard time delivering something other than software. That way software looks larger than a tool, it looks like a goal, even a purpose. But it is not. It is just a tool, which a software team builds to get its job “Make the life of the users easier” done.

I don´t want to belittle what developing software means. It´s a tough job. Software is complex. Nevertheless it´s just a tool.

On the other hand this view makes it easier to see how Antifragility and software go together. Since software is a tool, it can help or hinder Antifragility. A tool is a buffer. A tool has buffers. These buffers of cause serve Antifragility like any other buffers of a system.

So instead of saying, software should become antifragile – which it cannot. We should say, the socio-technical system consisting of humans plus software should become antifragile. A software company, a software team as organizations can have the property of Antifragility – or they can lack it.

A software producing organization can come under pressure. All sorts of buffers then can be deployed to withstand the stress. And afterwards the organization can decide to increase all sorts of buffers to cope with future stress in an even better way.

Antifragile consequences

µServices or an OR mapper or SOLID or event sourcing are certain approaches to build software. They might help to make its buffers larger. Thus they would help the overall robustness of the organization building the software. But the software itself stays a tool. The software won´t become antifragile through them. Antifragility needs deliberation.

Robustness of any order of a software is just the foundation of Antifragility. It can build on it, because robustness helps survival. It´s not Antifragility, though. That´s Nassim Taleb´s point. Antifragility emerges only if there is some form of consciousness at play. Stress and buffer depletion need to be observed and assessed. Then decisions as to how to deal with the resulting state of buffers have to be taken. Attention has to be directed to some, and withdrawn from others. And energy has to be channeled to some to repair them or even increase them. Which on the other hand means, energy has to be withdrawn from others. That´s why buffers shrink.

Shrinking buffers are a passive effect of energy directed elsewhere. Buffers are rarely actively deconstructed. Rather they atrophy due to lack of attention and energy. That´s what´s happening when you focus your attention on the rearview mirror while driving. You might be increasing your buffer with regard to approaching cars – but at the same time the buffer between your car and the roadside might shrink, because you diverted your attention from looking ahead and steering.

If you want to be serious about Antifragility as a property of a socio-technical system, then you have to be serious about values, strategy, reflection, efficient decision making, autonomy, and accountability. There´s no Antifragility without them. Because Antifragility is about learning, and liviing. Software is just a tool, and technologies and technical principles can only buy you buffer capacity. But who´s to decide where to direct attention and energy to? Who´s to decide which buffers to increase as a result to stress? That requires human intelligence.

First and foremost Antifragility is about people.

That´s why I suggest to drop the #antifragilesoftware in favor of something more realistic like #antifragilityinsoftwaredevelopment. Ok, that´s a tad long. So maybe #antifragileteams?


Wednesday, March 5, 2014 #

The notion of Antifragility has hit the software community, it seems. Russ Miles has posted a couple of small articles on it in his blog, asking what it could mean for software development – but also listing a couple of ingrediences he thinks are needed.

I´ve read the canonical book on antifragility by Nassim Taleb, too. And I second Russ´ description: Antifragile software is software “that thrives on the unknown and change”.

But still… what does it mean? “To thrive on the unknown and change” to me is too fuzzy for me to help me in my day to day work. Thus I cannot even judge if Russ is right when he says, micro services help to implement Antifragility.

That´s why I want to share some thoughts on the topic here. Maybe writing about it helps to clarify the issue for myself – at least ;-)

Is Antifragility something to strive for?

To me that´s a definite Yes. Yes, I want my software to be antifragile.

Clearly software should not be brittle/fragile.

That´s why there exists a canonical non-functional requirement called robustness.

But antifragility goes beyond that. Software should not just withstand stress, it should become better and better by being stressed. Software should improve because of strain, not despite it.

To be more concrete: Changing requirements – functional as well as non-functional – should cause software to be able to adapt even easier to further changes.

The status-quo seems to be the opposite: The more changes are heaped onto a software the more difficult changing it becomes. What we end up with is legacy code, a brownfield, a big ball of mud, spaghetti code. All these terms denote the same: a fragile code base. It´s fragile because the next change request might cause it to break down, i.e. to enter a state where it´s economically infeasible to further change it.

So, yes, I guess Antifragility is something we really need.

Fragility and robustness

But how can we implement Antifragility? How can we check, if approach A, tool T, concept C, principle P etc. benefit Antifragility or harm it?

I guess, we need to start with fragility and robustness to understand Antifragility.

What´s fragility? It´s when a small change in the environment causes harm. A glass vase is fragile, because a slight change in pressure can cause it to break. A football in contrast is not fragile. It´s robust with regard to pressure.

But maybe a football is not robust with regard to heat. Maybe it´s inflammable – and the glass vase not.

What distinguishes a glass vase from a football is their capability to compensate change of certain environment parameters. For short I call this capability a buffer.

If something is more robust or less fragile than something else, its buffers are larger. A glass vase and a football differ in their pressure buffers and their temperature buffers.

That means, if you want to make something more robust, then you need to increase its buffer regarding certain changes. For that you have to know the potential forces/stressors/attack angles. Do you want an item to be more robust regarding temperature or pressure or vibration or radiation or acceleration or viruses or traumata?

Into software you can build buffers for performance, scalability, security, portability etc. Software “can come under attack” from all sorts of non-functional angles. Today it´s 10 users per second, tomorrow it´s 1000 users. That´s stress on a software system. Does it have buffer capacity to compensate it? If so, it´s robust. If not, it´s fragile with regard to the number of users per second.

Antifragility

With this definition of fragility/robustness in place, what does Antifragility mean? Nassim Taleb says, it goes beyond robustness. Robustness is not the opposite of fragility, but just fragility on another level or less fragility.

I´d like to call Nietzsche to the witness stand. To me he´s the godfather of Antifragility, so to speak, because he says:

“What doesn´t kill me, makes me stronger.” (Maxims and Arrows, 8)

Yes, that´s what to me in a nutshell describes Antifragility.

Let me dissect his statement:

First Nietzsche acknowledges there are forces/stressed that can destroy an item. And I´d say, Nassim Talebl does not object. Antifragility does not equal immortality.

Destruction occurs if the buffer capacity of some robustness is exceeded. A glass vase might withstand a larger temperature change than a football. But in the end even glass melts and thus the vase form is destroyed.

That´s the same for antifragile systems. They have buffers, too. And those buffers can be stressed too hard.

But here´s the catch: What if the buffer capacity is not exceeded? What if some stress could be compensated?

For a more or less fragile/robust item this means, well, nothing special. It did not get destroyed. That´s it. Or maybe the buffer capacity is now lower. That´s what we call “wear and tear”.

An antifragile system, though, gets stronger. It changes for the better. No wear and tear, but increased buffer capacity. That´s the difference between levels of fragility and Antifragility.

Fragile, even robust items have static buffers at best. Usually though, their buffers decrease under repeated stress. Antifragile systems on the other hand have dynamic buffer which grow because of stress. Of course, they don´t do that during stress, but after stress. That´s why antifragile systems need rest.

Antifragile systems need time to replenish buffer resources – and extend their buffers. Try that out for yourself with some HIIT (high intensity interval training) for push-ups. It´s a simple proof for the Antifragility of your body. It might hurt a bit ;-) but you can literally watch over the course of a couple of days how your buffers grow, how you get stronger.

HIIT combines high stress (near depletion of buffers) and rest – and thus promotes growth.

And that´s what Antifragility is about: growth. Antifragile systems grow as a reaction to stress. Increasing stressed buffers happens in anticipation of future stresses. Antifragile systems don´t want to be caught off guard again, so they increase their robustness. In essence that´s learning. A larger buffer next time means less risk of harm/destruction through a particular type of stress.

At the same time, though, unused buffers shrink. That´s called atrophy. Stay in bed for a month and you´ll become very weak. Not after a night in bed, not after a day or two. But prolonged underusage of muscles causes them to dwindle.

That´s just economical. Keeping muscle power buffers large costs energy. If those buffers are not used, well, then why spend the energy? It can be put to good use at other buffers or saved altogether.

Antifragility thus does not just mean growth, but also reduction. Stripping down buffers is nothing to fear for a truely antifragile system, because it can build up any buffer at any time if need be.

This (meta-)capability of course is also limited, however. It´s a buffer like any other. And it can be grown – or it can shrink. That´s what we usually call aging. A system ages if it loses its Antifragility. In the end it´s just robust – and finally worn down by some stress.

Becoming alive

If we want to talk about Antifragility in software development we need to talk about life. Software needs to become alive. Or the system consisting of code + humans needs to become alive. Truely alive.

We need to identify stressors. We need to identify buffers to compensate those stressors. We need to measure the capacity of those buffers. We need to find ways to increase – and decrease their capacities timely and at will in reaction to stresses. We need to assess risks of stress recurrence in order to determine if increase or decrease is appropriate. We need to rest. And finally we need to accept death.

To me that means, Antifragility is much, much more than SOLID code plus some micro services plus message based communication via busses plus reactive programming etc. First of all it´s not just about code, because code itself is not and cannot be alive. Thus code itself cannot change itself for the better after stress, like no football can increase its temperature robustness.

Rather the “socio-mechanical” system made up of humans and their code needs to be formed in a way to be antifragile. Which of course also implies certain beneficial structures and relationship in code. But that´s not all to think about.

So we´re not there, yet, I´d say. Antifragility is more than Agility + Clean Code. First of all it´s a mindset, a worldview. And as usual, to instill that into people is not that easy. It will take another 10 to 20 years…

Therefore let today be the first day of another journey in software development. Let the Antifragility years begin…


Wednesday, February 12, 2014 #

In my previous article I described a process of designing and implementing software. It combined TDD with explicit thinking before coding. The dialog I embedded it in was supposed to make this deviation from traditional descriptions of TDD more palatable. This came at a price, though. The systematic behind this approach was somewhat hidden. This article is supposed to make up for that. I want to make it crystal clear how I think the process of solving problems with code should look like.

“Traditional” TDD

But first a recapitulation of how TDD “traditionally” is depicted. I´m not saying this is how its inventor(s) thought it should be done. Whatever they had in mind, though, got reduced to this:

image

For a given functional unit of production code you write a test. It´s red/failing first. Then you do the simplest possible thing in production code to change it to green/passing. Finally you refactor.

Sounds simple. And it is simple – technically. The hard part of TDD is in the, well, details. Or in the cracks between these steps.

Question number #1: Where do the test cases come from? From the customer? Hardly at the level of detail that´s needed to drive out a solution from the inital ignorance. The customer of course should provide acceptance test cases. But they are coarse grained. What´s needed to refine the solution in small steps are fine grained test cases. And they should be in a certain order so that the solution code really can become progressively more complicted.

So where do those nicely prioritized fine grained test cases come from? That´s rarely explained in the TDD literature. Authors try to derive them from the problem itself or maybe from some acceptance test cases. But never have I seen test cases being linked to the solution. The reason for that is simple: the solution is supposed to somehow emerge.

You write a failing test, then switch to the production code side, and there you just do the simplest thing possible. Solution follows test case. That´s the “traditonal” TDD mantra.

Unfortunately I´ve seen many, many developers fail at that. They make quick progress – with obvious and trivial test cases, so called “degenerate test cases”. And then they hit a wall.

Try this for yourself with the famous Roman Numerals code kata. It´s about a function (e.g. string ToRoman(int number)) to convert arabic numbers like 5, 19 or 42 into roman numbers: V, XIX, XLII.

How does your list of test cases look like?

Let me speculate: 0, 1/I, 2/II, 6/VI, 4/IV, 9/IX, 40/XL, …

First the degenerate test case: 0. Because there is no zero in the roman number system. You need to check for this input, though. Don´t you? That´s good defensive coding practice.

Then the simplest roman number: I. It consist of just a single character.

Then a sequence of the same character/number. A sequence of like values is “higher order” than a single value.

Then a sequence of different characters/numbers. A sequence of different values is “higher order” than a sequence of like values.

And now for the fun part: the first of those pesky numbers with “reversed digits”. Usually “larger” roman digits precede “smaller” ones like in XVI. But some numbers require for a reversal of this order like in IV or IX or XL.

Given the premise “solution flows from test case” this would be a perfectly reasonable order and granularity of test cases. At least that´s what I´ve seen many times. And you can find it in the literature, too.

In reality, though, – as is often the case compared to technical or recreational literature – things work out differently. And so it happens more often than not, that developers doing the Roman Numerals kata like this to exercise their TDD skills fail. They do not arrive at a working solution in a moderate time frame of, say, 60 minutes.

Of course several excuses are readily at hand. I´ve read and heard often for example “We wanted  to have fun. So we did not really focus on a solution.” or “We wanted to focus on the red-green-refactor steps. A working solution is not that important.”

That´s all nice and well. I don´t want to deprice anybody of his or her fun with TDD or at their dev community gathering. Also I fully understand that in order to learn A you sometimes need to sacrifice B and C for a while. Otherwise you can´t fully focus on doing A right. In the end, though, that means in your daily work practice, A + B + C must come together.

But, hey, come on. What are we talking about here? Roman Numerals is a trivial problem. It´s orders of magnitude simpler than anything you´re doing for profit. It´s like drawing a stick figure compared to the Mona Lisa.

No, I don´t buy “focussing on the TDD steps” as an excuse for not solving a trivial problem in an expanded timeframe like 60 or even 90 minutes. That´s inacceptable for any method. And it neglects the need for closure. Developers want closure. They want to accomplish something, that means they crave for building something that works. Failing to satisfy this need hampers any learning.

And finally: Even if working solutions are produced, I rarely have seen any which really have lived up to the full TDD process consisting of three steps. The refactoring step often is skipped altogether. “We can refactor later. First let´s get this baby off the ground. We need working code.”

Most TDD solution on the internet don´t show any sign of refactoring. Their design is… well, “as usual”.

If you think about this for a minute, this is very plausible for two reasons. Firstly, closure is reached if the functionality is implemented. Clean code is not needed. Secondly, where should a good design come from? TDD does not contain any hints on good design. It just commands: “Thou shalt refactor once the test passes.” But as is with most of such commandments, they are overheard in favor of some seemingly more important issue.

And since rules for good design are orthogonal to TDD, refactoring is neglected except for fairly trivial cases.

Please get me right: I´m not against TDD. Red+green+refactor are great – if they only produced the results the claimed on a satisfying scale.

So this is why I´m looking for a different approach. Can the “traditional” TDD approach be improved. And I think it can. One way to improve it has been described by Keith Braithwraite as “TDD as if you meant it”. It improves on the refactoring step. With it refactoring becomes inevitable. Great!

Another way to improve the TDD rhythm is by thinking before coding:

Informed TDD

My experience is, software development becomes easier if we think before coding. Sounds trivial. Sounds like something everyone´s doing already. But I beg to differ.

We´re far to quick to grad our keyboards and start coding. The lack of working code for the “stick figure problem” Roman Numerals after 60 minutes to me is proof to that. If experienced developers are unable to code a solution – with or without TDD – in that timeframe, then something is wrong.

What´s missing is not good will, motivation, technical skill. It´s experience in conceptualizing solutions. Systematically thinking problems through to arrive at solutions or at least reasonable (rough) ideas of solutions is an art not valued – at least in many developer circles I´ve attended in the past years.

Everybody is trying his/her best. Of course! There is no lack of motivation. However, the hard work put into software development sometimes resembles a prison inmate trying to dig his way out with his bare hands. It would be much easier for him had he a pick ax, or even some explosives at his disposal. Or better: a key. Or even better: not be in prison in the first place.

But to someone sitting in a whole these don´t seem to be options. That´s why we need an explicit emphasize on thinking. We need preventive measures. Before developers convict “the crime of premature coding”.

For this article I want to call what I´m trying to bring across “Informed TDD”. This is to underline the value of TDD – while at the same time adding a missing piece, namely information. Information about the solution to be coded.

Ok, here´s how Informed TDD (ITDD) differs from Traditional TDD (TTDD).

image

Like TTDD I start with a functional unit. But instead of writing a failing test right away, I think before coding. The result of such thinking is a solution approach. That means I develop an idea of how the problem can be solved using “custom tools”. These custom tools are special “code machines” to be arranged in a transformation process. Think of it like an assembly line in a factory.

“First this is done, then that is done, finally something else is done.” - that´s the most simple form of an idea for a solution. Yeah, it´s simple – on the surface. But it´s powerful. Because what I get are smaller problems. And if I repeat this process for the resulting “custom tools” I get even smaller problems to solve.

This is called stepwise refinement. Divide a bigger problem into smaller, simpler ones. By thinking, by simulation, by imagination, by drawing on a whiteboard, maybe by experimenting with some code.

The two leaf-circles represent the function units for the “custom tools” somehow integrated into a whole by the initial functional unit, the root.

Next comes a red test. But this now is guided by my knowledge of the outline of a solution. The test case can be very different from one that you´d choose doing TTDD.

image

The interesting question now is: How to get the test to green?

View it as a hypothesis. It states: If the function sub-units work like supposed, then the system under test (root) is correct. Because the job of the system under test is just to integrate those two leafs. Nothing more. That´s the Single Responsibility Principle (SRP) taken to an extreme.

How can this hypothesis be verified? By faking the implementation of the leafs. Using a mocking framework like JustMock that´s easy even without IoC/DI.

image

Now I´ve two simpler problems to solve. Of course I do that the same way as before. I think, before I code. The focus now is one of the leaf functional units.

This time I don´t see how to further refine it, though. There are no “tools” I can think of. Just an “algorithm” I can code. In that case I´m back to TTDD.

image

However the leaf probably is a private method hidden behind the public root method. That´s why special measures need to be taken to make it teastable. A framework like JustMock helps here, too.

If needed, I add another test or refactor. But in the end I go back to the root test and remove the mock:

image

Now the hypothesis is proven not only by mocks, but also by production code. That´s better.

Next I need to tackle the second leaf of the root. ITDD tells me to think first. Can I come up with smaller steps to reach the goal of the whole leaf? Yes, this time I can:

image

That means I need to continue according to ITDD: Write a red test, then replace the sub-leafs with mocks.

image

And so the process continues “into the deep”. I´m drilling down into the ever finer details of the solution.

Tackle the first leaf with ITDD:

image

Tackle the second leaf with ITDD:

image

Then backtrack to the top and remove the mock there:

image

And finally… remove all non-root tests. They are not needed anymore. It´s just temporary scaffolding while fleshing out the solution. The essential test is the test of the root functional unit.

image

That´s it. That´s what I mean by Informed TDD.

Technically it´s easy – with a little help of a framework like JustMock or Typemock Isolator (both for .NET).

The benefits of ITDD I´ve come to value are:

  • Refactoring is less necessary, since there is a clean design before I start coding.
  • Testing of solution aspects does not need to go through the root all the time. That makes it easier on the test cases.
  • Test cases are less contrived because they are informed by a solution. This leads to less groping around for the right code.

Also I find this process natural compared to other “manufacturing tasks”. I you build a table from wood or repair your kitchen sink you mess around and clean up. The surrounding of the “work item” loses and regains order. That´s what´s happening with ITDD too. Tests are written, mocks are created (mess) – and later discarded (clean up). I don´t need to be careful with my tests or at least not with all the tests. Some tests have a long lifetime: acceptance tests. But others are just temporary.

I hope this helped to clarify how I think TDD can be enhanced by a little explicit thinking before coding.


Monday, February 10, 2014 #

“Hey, Ron, wanna try something new”, Janine asked across the table in their team room. Ron looked at her uneasily over his line of monitors. For hours he had been trying to fix this elusive bug.

“Nah, not now. I really need to get this done first, Jan.”

“Oh, come on, Ron, that´s what you´re telling us all the time.”

Ron blushed. That certainly wasn´t true. Especially not with regard to Janine.

“You know me better than that, Jan. I´m always willing to help you out and stuff. But right now…”

“It won´t take long, Ron. Promised. It´s just so cool. You´ll like it. And… maybe it even helps you with your current problem. Who knows…” Janine winked at him – and of course Ron´s resolution to not again get distracted by her crumbled. He sighed and rolled his chair next to her.

“Ok, what´s it? You better make it quick, Jan – and exciting.”

Janine beamed at him. Then with a secretive look and a hushed voice she said: “I found a better way to approach coding. We´ve been doing TDD, right. We´ve tried hard to do it right, to do it by the book.”

Ron nodded slowly. What was she trying to get at? For the past couple of weeks they had run coding dojos with their team to get up to speed with TDD. Finally. He strongly hoped she wouldn´t suggest to deviate from that course. It had been difficult enough to convince their colleagues to at least try automated tests. Those guys were old code cowboys and not so happy to be asked to learn new tricks.

“Oh, Ron, don´t give me this skeptical look again. Please. It´s not what you might think. TDD is cool – but we can do even better. Remember how we had a hard time doing the kata Roman Numerals? We tried hard, we moved along in tiny steps – but in the end of our timebox we had not finished the task.”

Yes, he remembered that particular coding dojo very well. Their colleagues had been very frustrated.

“Sure. But our retrospective made clear what we´d done wrong, didn´t it? It was a matter of wrong test priorization.”

“That´s right, Ron. But why did we get the test prios wrong? Again, that is. Because we didn´t give it much thought. Because we did not think about the solution first. We were so eager to get coding…”

“Well, only in code lies truth, Jan. You know that.”

“True, Ron, true. In the end at least. But what code to write? Where should the code containing all the truth come from? How do we transform a test case into code?”

Ron stared at her blankly. What was this? A lesson in philosophy? Programming is about detail. Code represents all those details. Details need deep technological knowledge. Without solid craftsmanship no running code would get to their customers.

“Come on, Jan. Don´t play this kind of game with me. You know I hate it. Just tell me what you want to tell me.”

Janine sighed. Usually Ron was very pleasant to work with. A guy who liked to explore the unknown like she did. But sometimes… She took a deep breath, turned to her monitors and pointed to a C# file.

“Ok, Ron, see those test cases? It´s the kata Word Wrap. We did it a while ago. Tried to do it like we saw it on the web in Robert C. Martin´s blog. And then – you remember? – George came up with this additional test case. And our solution did not handle it correctly. Neither did Martin´s show case solution.”

Ron nodded. That had been one of the more successful coding dojos – well, until they found out, TDD had not led them to a bug free solution right away.

“I have pondered long about why in the end we failed. And I guess I´ve found the root cause. We did everything right TDD-wise. But we did not enough thinking upfront.”

“What? We should think more ahead? Big design up-front, Jan? You can´t be serious. Our retrospective clearly showed we just failed to come up with the additional test case. Had we added it to our list we surely would have also solved that problem.”

“Yeah, you´re right, Ron. But how can we think about test cases in the first place if we don´t have a clue about our solution? Test cases not only depend on the customer, but also on how we design the code.”

“Yes, the design naturally flows from the test cases and is refined through refactoring.”

“Ron, just listen to what you´re saying. I know, this is how we´ve thought so far. But can this be true? How can code flow from test cases? Code, and its design are the result of our thinking. We plan code before we write it. Maybe we´re not aware of it, but there´s no other way. It is born in our heads. Before we code. The questions are just: When? And how explicitly?”

He took a deep and somewhat resentful breath – but before he could retort, Janine drew him back to the code.

“I know, this sounds theoretical and somehow against what we´ve read about TDD and stuff. So let me show what I did. I read a blog post about an alternative approach to test-first programming and tried the suggestions out. Just follow me for a couple of minutes, will you?”

Ron gnashed his teeth, but kept silent and noded.

“Ok, thanks, Ron, honey.”, Janine said and smiled at him disarmingly. “I want to do the kata Word Wrap with you in a different way. These acceptance tests are our starting point. Currently they are all red. Of course. The Wrap() method just throws a NonImplementedException. I want to flesh out its implementation together with you.”

 

image

 

“It´s all the canonical unit test cases – including the one which brought our prior implementation to its knees. And two more test cases with real texts. Just for the fun of it. Now, what I want to do differently is: I want to think about our solution approach first.”

“What do you mean by ‘thinking’? Do you want me to draw an UML diagram, or what?”

“No, no, Ron. No UML. You know I hate that, too. I mean, well, just ‘thinking’ or talking about how we would accomplish the word wrapping task by hand.”

“By hand? We need to code, Jan.”

“Yes. In the end we need to code. But software always does what we as humans could do by hand or in our heads. Just much faster. It´s the non-functional requirements software is written for. Functional requirements can always be fulfilled without software – given enough time and money.”

“You mean, functionality is not important?”, Ron exclaimed in disbelief.

“No, of course. It´s essential. But… Oh, no, let´s no digress. We can talk about that later. Let´s focus on the kata for now. How could we do ‘wrapping words’ by hand?”

Ron stared blankly at her. After a short pause Janine took a piece of paper and a pencil and wrote:

1. Split text into words

2. Build lines from words

“How about that, Ron? Wouldn´t that very simply describe how word wrapping could be done?”

“Hm… yeah, sure. But that´s not code. It´s just words. You don´t know if this is actually working.”

“Oh, Ron… Of course this is just words. But those words can be translated into code, can´t they? We do that all the time. Look here.”

 

image

 

“This is not just words, it´s even bubbles, Ron. I know, they don´t crash. But still it´s a representation of our code. It´s like a little plan: We intend to write a functional unit which transforms an incoming text into an outgoing text with line breaks. It´s like a map of our code. It´s not the code itself, just an abstract view. The same is true for what I´ve written above. It can be depicted like this.”

 

image

 

“I just refined the first functional unit. The finer grained functional units describe how the mission of the coarse grained unit can be accomplished. You know how to transform the coarse grained unit into code – you define a function. So why not do the same with the finer grained units?”

“What do you want me to do, Jan? Write three functions without a test? Is that your new shiny road to reliable code?”

“No, Ron, not that quickly. First I want you to see that thinking brought us closer to the solution. Instead of a big problem, we´ve two smaller problems. And we have a simple hypothesis: if there was a function ‘split lines into words’ which did what´s suggested by its name, and there was a second function ‘build lines from words’ which did what its name suggested, then their combination would solve the word wrap problem. Yes, it´s just a thought. But one that can easily be verified. By code. Give it a try. Write a simple failing test for the Wrap() function. I´ll then take over and show you what I mean.”

At the prospect of writing code Ron relaxed visibly. He gently pushed aside Janine on her chair and grabbed her keyboard. In no time he set up the following code:

 

image

 

“Thanks, Ron. I see you´re cutting right to the chase. You´re right, we don´t want to waste our time on unenlightening trivialities like tests for null or empty strings. Now let me show you how I want to turn this test to green. Currently it´s failing because there is no solution code behind the Wrap()-facade. Let´s change that first.”

Janine took over the keyboard and fleshed out the Wrapper class like this:

 

image

 

“I hope you see I translated our solution idea into code. The job of Wrap() now is to just integrate two functions into a larger whole. How those functions accomplish their sub tasks I don´t know yet. Before we turn to that let´s check our hypothesis. Let´s turn the test green.”

“How do you wanna do that, Jan? Those functions both throw exceptions.”

“Well, Ron… let me apply some very, very advanced technology here – which is indistinguishable from magic, as you know“, Janine said grinning while mimicking Hermine wielding her magic wand. “But to be honest, I just downloaded a cool mocking framework called JustMock from Telerik.”

 

image

 

“This test proves: if those two functions worked correctly, the whole would work correctly. You could say: we just fake it until we make it.”

Surely the introduction of a new tool grabbed Ron´s attention. Regardless of Janine´s strange approach to programming this was something to take a closer look at. Maybe she was right and this little breakout session with her could help him fix the darn bug.

“Jan, even though I don´t know if I like your approach, I´m amazed at how this mocking framework of yours can replace even private methods. How about methods from .NET Framework types or even whole types? Have you tried that?”

“No, Ron, I haven´t. And please let´s not get carried away here. I know it´s a cool tool. And you can check it out later. For now please stay with me. I want to show you how I think this technology can help us become better at TDD.”

Ron moved a bit restlessly in his chair, but Janine was able to regain his attention by asking him to write a next test.

“Since this test is green, why not move on? We need to tackle one of the smaller problems now. This approach I want to show you is recursive. TDD is applied on every level. So why not write a red test for Split_into_words()? Ron, your turn again.”

Ron looked puzzled. “But, Jan, those methods are private. How could I write a test for them?”

“Oh, Ron, you baby, don´t you trust our new shiny tool? Here let me show you.”

 

image

 

“But you´re not even supposed to test private methods. They are details hidden behind the public interface of a class for a reason.”

“That´s true, Ron. But as you´ll see, we won´t violate this rule for long. Come on, get this test to green. Be a good craftsman for me, will you?” And so Ron dug into his knowledge about string manipulation with C#.

 

image

 

“Great! It´s simple, yet pretty comprehensive, I guess. This not only covers the happy day case of the test, but also a few other scenarios. I guess, as long as we keep our requirements constant, that words are separated by spaces, this will do. So let´s go back to our hypothesis and retest it – but now with the real functional unit instead of a mock. Can you imagine what I want to do?”

Ron frowned. But then his eyes opened with near despair. “Oh, no, Jan, you´re not gonna change the previous test, are you? You´re not supposed to do that.”

“Yes, Ron, that´s exactly what I´m gonna do”, replied Janine with her well known comforting voice. “Why not change a test? It´s nothing sacred. Esspecially not these ‘under the hood’ tests. They are all just temporary skaffolding while erecting our code building.”

 

image

 

“The test got simpler. And now it´s even more valuable, because our hypothesis is even true for some real functionality, not just fake bubble implementations. And now let´s repeat these steps for the second processing phase of our solution. Your turn again, Mr. Craftsman.”

Ron turned to Janine´s keyboard and applied his newly won knowledge of JustMock:

 

image

 

“This sure is a non-trivial test, Ron. And I´m eager to see, how you´re gonna get it to green,” said Janine smiling in a way that made Ron feel a little strange.

“Well, I choose the simplest possible implementation first.”

“Yeah, but what´s that? Do you want to return the string ‘a b’?”

“No, Jan, of course not,” said Ron a little unnerved. “I´ll actually solve the problem of joining words into a line.”

 

image

 

“Oh, how clever of you, Ron!” Janine acclaimed jokingly and clapped her hands. “That get´s us to a very cute light green. But not more, you know. Because it does not even try to use the second parameter.”

“Sure not. The test case does not require it. That´s the point about making test cases progressively more complicated.”

“You´re right, Ron. About the test cases. But why make the solution overly dependent on the test case? Why eschew the opportunity to really learn something about the problem domain? The second parameter is there for a reason. Admitted: if it wasn´t there, this would be a good first try. But we´ve design the function with a second parameter for a good reason. It´s the workhorse function of our solution to the word wrap problem. So we need the second parameter. That´s not looking into a crystal ball, but solid untderstanding of the problem at hand. But anyway. Leave it as is and move on to a second test. I guess we´re agreeing at least on that we need one more.”

Ron wanted to reply – but then settled for doing as asked. That way he´d surely get back to his computer more quickly for some real exploration of JustMock.

 

image

 

“Yes, that´s what I thought of, too. Now watch what happens to your previous solution…”

Ron switched to the implementation of Create_lines_from_words(), pondered the one-line solution, frowned – and then his eyes lit up.

“Ah, I see what you mean. I have to throw it away. Right? There´s nothing I can carry over from the previous test, because now the second parameter becomes important, but I did not use it.”

Janine nodded and smiled at him broadly. “You got it, pal.”

“So let me fix this first. I´ll replace the trivial solution with one that´s actually tackling the problem – but still in a simple way. How about this?”

 

image

 

“That looks much better, I guess. Even though we know the maximum line length is never exceeded, this code demonstrates an idea of how it can come into play. But let´s not dwell on this for long. We´ve to solve the new test case.”

“Right, mam, at your service,” Ron said jokingly and continued implementing the function.

 

image

 

“This does the job, Jan. But I´m not 100% content with it. Looks a bit clunky, doesn´t it?”

“Yeah, I guess so. But then… it´s just some 20 lines of code. The purpose is pretty clear. Important terms from the ubiquitous language are present like text, line, max. line length. You even extracted a method for adding a finished line to the text.”

“That was an experiment. I saw nested function definitions in F# and liked it. So I thought about how to mimick that in C#.”

“Pretty nifty to use a lambda expression. Yeah, why not. Let´s leave it at that. We can always come back and refactor more or even find a better overall approach. But first we want to go back to our hypothesis…”

“You want to remove the second mock?”

“Exactly! Then we´re truely gonna see, if our hypothesis was true.”

 

image

 

“Ah, finally! This looks good. It´s still green. So we were right, Ron. Now, let´s check how many of our acceptance tests have also turned green.”

 

image

 

“Hey, look at this! More than half of the acceptance test cases are already fufilled by our solution. Not bad for a start. And the non-canonical test case which crashed our coding dojo solution is among the green tests. That´s especially great because we didn´t labor much about the individual cases. Remember how we discussed where and when to break lines? But now, with just a little thinking before coding, with an idea of a conceptual solution, we just covered it.”

“That´s true, Jan. Maybe this approach of yours isn´t that bad in the end.”

Janine blew Ron a kiss.

“But don´t go overboard now, Jan! The devil is in the detail. We´ve three more tests to go.”

“Ok, ok. Let´s move on. We need to tackle the next category of tests: what about words longer than a line? First a red test. Your turn again, Ron.”

“That was the category we started the coding dojo with. But I guess it´s more natural to do it second. The probability for such long words is small – at least in the real world, I suppose. So it´s a special case.”

 

image

 

“The test was easy to write. But how should we go about to make it green, Jan? You sure have your own ideas about that.”

“I sure have, Ron. Because we´re again in a situation where we need to find a solution first, instead of quickly slinging code. The current implementation does not cover the test case. What´s our idea of how to tackle wrapping long words?”

“Hm… as far as I remember they may be cut off at any point. There are no special requirements for something like hyphenation.”

“Right. That makes it easy to enhance our current solution, I presume. How about adding a processing step between the exiting ones? That way we would not need to change existing code, except the simple integration function Wrap().”

1. Split text into words

2. Split long words into “syllables”

3. Build lines from words

“I call the split words ‘syllables’ because that´s what they look to me, even though we won´t use hyphens at the end of lines. At least for now.”

“Sounds good to me. Makes for a precise term of our ubiquitous language. However, is it ok to split all long words into syllables, Jan? In the usual kata solutions words are split at the end of lines, only. But this approach will do it proactively.”

“I don´t find the requirements very clear about this aspect. So I assume we´d be within the boundaries of what´s permissible. And in the end at least the canonical acceptance test cases will tell.”

“Ok, let´s try it this way. Now, let me see if I understand your approach. Is this how you´d enhance the code: put in a function for the processing step and first mock it to get the test to green?”

“Right on, Ron! That would make our hypothesis tangible.”

 

image

image

 

“The mock-up nicely plays together with the other functions. The test is green. Now we´ve to drill down and focus a test on the new function Split_long_words(). You wanna do that, Ron?”

“Sure. Especially since the first test again is the easier one.”

 

image

 

“Yeah, I see, you´re going for the low hanging fruits, Ron. But you know: the test is only half the fun. I want to see you sweat over a simple, but not too simple implementation. Remember: don´t eschew solving the problem.”

“I saw that coming, Jan, I have a plan already. Take this.”

 

image

 

“That looks great, Ron!”, Janine said nudging Ron jokingly. “What´s up next?”

“Well, I guess a test for truely long words. But guess what: Even I can now see that´s a task calling for a function of its own. A function to split a single long word. And that needs to be mocked first.”

Janine´s jaw dropped as she looked at Ron in disbelief. Was it possible he already had gotten the knack of her new approach? Yes, it seemed so. How cool was that…

“Yeah, I know, you thought, I would be more skeptical. But I can feel what you mean. This really is a helpful way to do TDD. It´s small steps. But these steps are better guided. Guided by thinking. I like it.”

That said Ron showed Janine how he thought they should continue. First a red test using mocks:

 

image

Then the enhancement in the production code:

image

 

This turned the test green. Another hypothesis formulated.

“That feels like a systematic process, doesn´t it, Ron? If stuff is more than trivial, then the test data is not enough. Some idea of how the expected result is supposed to be created is in order – and should be encoded in the test.”

“Yes, the mocks are representations of a design idea – which needs to be validated.”

“And this is true on any level of abstraction as we can see. We did it like this on the top with Wrap(). And we´re doing it again down here.”

“Yep. But with each level it becomes easier. The functions become smaller. Look at my stab at this new function. A test first, then an implementation.”

 

image

image

 

“Impressive, Ron, impressive. You wrote that down in no time. Had it all in your head? Oh, you´re my hero coder of the day…”, Janine purred and Ron blushed. He knew she was joking, but still… There was something about Janine, he liked very much. At least as long as she didn´t try to talk with him about some of her even more esoteric ideas on software development. Like event sourcing or reactive something.

Then, when Ron opened his mouth after a tiny, but uncomfortable pause, Janine threw up her arms and called out loud: “Look, Ron, we´re done! All tests are green! All of them!”

Ron swallowed a supposedly funny remark he was about to make about him being her hero and looked at the code. Indeed, all methods where shown with a green dot next to them. That meant, NCrunch, the background test runner, had found no failing test.

 

image

 

“Wow. Awesome! That means all our acceptance requirements are fulfilled. And that means, our approach to splitting long words can´t be all that wrong.”

 

image

 

“Well, what can I say… So, cool Ron! Your implementation did it. We don´t even need to remove the mocks on the higher levels. The proof is in the acceptance tests.”

Nevertheless they took out the mocks and found their hypothesises validated.

“Jan, I need to say…”, but Janine interrupted Ron. “There´s one more thing I want to show you, Ron. Watch, what I´m doing now. We checked in the current state of the code. Right? Ok, here we go…”

And Janine deleted all the unit tests.

Ron nearly fainted. All their precious work gone…

“What are you doing, Jan?!”

“I´m cleaning up, Ron,” said Janine and smiled broadly. “Clean code does not mean it needs to be clean all the time. We can – and should – make a mess while working on the code. In the end, though, once we´ve reached our goal, we need to clean up. That also means to throw away any scaffolding code or any code we produced along the way while ‘grinding’ our implementation.”

Ron wiped some sweat from his brow. This girl really knew how to shake up his convictions.

“Look, Ron, this is like building a house. While building it, you make a mess. You set up all sorts of additional structures to easily produce the building. Think of how the Egyptians build the pyramids. Today you only see those geometrical shapes – but while they were building them, there were all sorts of ramps. Otherwise it would have been impossible for them to drag those heay stones up to the top.”

 

image

 

“Or think of the last time you made dinner for someone. You surely made a mess while cooking, no, even in order to cook. But in the end, there was just cleanly layed out food on the plates. All the mess gone into the trash can or the dish washer.”

Whether this was true, Ron couldn´t remember at the moment. Janines images evoked not so nice memories of the final period of his marriage. His wife and him trying to mend their relationship by doing more stuff together like cooking. But, alas, to no avail. It even seemed that “pair cooking” upset them more than bringing them back together.

"…but just chippings. – Ron? Hello?”

“Yes, what? Oh, I´m sorry. What did you say?”

“I said, we´re not throwing away precious code, but just chippings. Or molds. That´s perfectly fine. It makes for better focus. And look at the test coverage: it´s still 100%.”

 

image

 

“Shedding all the detailed tests also makes it easier to change our solution later. We don´t need to be careful to not break any tests, but can refactor boldly – as long as the acceptance tests stay green. And if we need to zoom in on a detail, e.g. if we want to change how long words are treated, we (re)create any tests we need.”

Ron nodded. This was quite a bit of input for him. At he beginning he had be skeptical about a new approach to TDD. Then he got the knack of “thinking before coding”. And the mock framework was fun. But trashing all their test work… he wasn´t convinced yet despite all the effort Janine invested.

“Jan, this sounds good, when you describe it like this. However, I guess I need some time to digest this experience.”

This saddened Janine a bit. She had the impression, Ron had jumped on her bandwagon. But then… the capacity to absorb changes to one´s beliefs is different from person to person. Maybe she really had been too enthusiastic at the end.

“Ok. I´m sorry if I blew it. I just was so excited about how well this approach was working, and about JustMock, and all. You know, I thought this really could spark up some new experiments in our coding dojos.”

Ron smiled at her a tad tired.

“Yes, Jan, we´ll try it out next time with Dave and George. But now I need to go back to my bug. It probably hasn´t fixed itself in the meantime. But, hey, thanks for calling me over. This was fun. And I´ll check out JustMock. I have a hunch it can help with the bug.”

“Alright, then, Ron. Thanks for listening to my babble. It was good to go through this again by telling somebody. Helped to clear up my thinking.”

Ron was about to turn around his chair when Janine got up and very lightly hugged him. Then she quickly turned around and barely audible added “Thank you, Ron.”


Friday, January 31, 2014 #

I read, with interest, Robert C. Martin´s comment on Justin Searl´s critique of a common TDD teaching approach. Strange it was to see “the good uncle” to be so upset about Justin´s article. Why does he take it so personally, it seems, that someone is not content with the state of TDD-affairs? Why salting Justin´s earth?

Anyway… what I wanted to express is my disagreement with Robert C. Martin´s idea of a “Domain Discontinuity”. He proposes there to be two levels in software system design: one, where thinking before implementing is good, and another one where it´s bad. The first being architecture, the second being “the problem domain”.

Unfortunately, he does not define what he means by “architecture”. And his definition of “problem domain” is shallow at best: business rules.

But why this dichotomy?

It´s the requirements, stupid!

Strangely the term “requirements” occures only once in Robert C. Martin´s article. He associates it with tests. I completely agree with that. But why stop there?

Each and every trait of a software system – the look of the UI, the performance of some calculation, the distribution of code across clients and servers, the choice of persistence technology, the number of classes, the interfaces of those classes etc. – must be traceable to some requirement. This is true for coarse grained traits as well as fine grained traits.

The basic categories of requirements to me seem to be:

  • functional requirements or functionality for short
  • non-functional requirements or (primary) qualities for short, e.g. performance, scalability, security, usability etc.
  • evolvability, i.e. a special quality regarding the flexibility to adapt the software system to new requirements
  • production efficiency, i.e. a special quality regarding the ease and speed to implement requirements; it´s a quality of the development process

Software development thus is the art (or craft or disciplin) of making the right choices from a set of options to best fulfill the concrete requirements falling into those categories.

Architecture for qualities

With requirements put first the question again is: What´s software architecture in particular about?

My simple answer to this perennial question is: Software architecture is about designing structures to fulfill non-functional requirements aka qualities.

I don´t care much about whether architects need to be top communicators in order to do this job. I don´t care much about whether they need to be top notch coders to do this job. I don´t care about their tools or the level of their architecture documentation. Just do whatever is necessary to transform requirements into “ideas” of structures for code (and other artifacts).

Architects don´t build, they design. That means they think – before someone else implements. (Please get me right, when I say “architect” I don´t mean a person but a role. The same person can be an architect at one moment and something else in the next moment. I do not advocate special architect job positions. Right to the contrary: I think each and every developer should hone his architecture skills.)

Sure, architecting software is not easy. Not because so many paradigms and technologies need to be taken into consideration, but because of the need to balance. Qualities are contradictory at times. They can pull software structures into opposite directions. Designing for performance might impede evolvability. Designing for security might impede performance.

Architects thus need to deliberate much. They need to think.

Yes, not too much. I know. BDUF´s still bad. But some thinking is in order. All else would negate thousands of years of engineering success. Where the requirements are clear, up-front thinking leads to a solution – however tentative that might be.

So when does thinking about qualities end? Does it end after choosing MVVM over MVC and encapsulating MongoDB access with some kind of adapter or moving business logic into stored procedures?

I won´t say so. Because architecture is whereever anybody decides anything with regard to qualities, especially when it´s pertaining to the software´s structure. That means, architecture is at work while (!) doing TDD. Because what drives the refactoring of a solution? It´s some quality, not functionality. Why extract a method, a class? Why segregate interfaces? Why follow IoC and do DI? That´s all architectural questions.

There´s nothing sacred or lofty about architecture. It´s simply about a couple of issues that need to be resolved, quality issues. Anybody with the right knowledge and capabilities can be an architect. In fact every developer is every day.

Functionality for the problem domain

The second term Robert C. Martin uses is “problem domain”. With the above categories of requirements I guess I know what he means by that: functionality. Not quality, but functionality. That´s fair enough.

And he says, tests should be used to formalize functional requirements (i.e. desired behavior) before the implementation. I agree with that, too.

What I don´t get, though, is: Why not continue to apply the human brain afterwards? Or even before? Why devalue thinking in the form of “thinking up a design”?

If (functional) requirements are clear enough to hard code tests, why aren´t they clear enough to at least come up with a solution sketch for the part of the problem domain in focus?

Again and again I have seen experienced developers fail at simple problem domains like “kata roman numerals”. They don´t fail because they lack extraordinary TDD skills. No, they fail because they have little practice in actually “thinking about solutions before coding”. Theír tool chest contains only a single tool: code.

So what to do about refactoring? Well, where should an idea of a better structure come from when you´re not used to thinking before coding? Refactoring as the third TDD step just defers thinking to the last possible moment – and beyond, as the many TDD examples show where not much refactoring was done. Refactoring is optional. Nothing forces refactoring in TDD. That´s why Keith Braithwaite came up with “TDD as if you meant it”.

But why then say, “They just haven´t gotten it! They need to learn to do better TDD!”? Why not question for a moment the “traditional” TDD approach? Why not – as Justin does – introduce at least some thinking before coding?

The cleanliness of code which TDD might aim at also is just one of many qualities – which needs to be balanced against others, like production efficiency.

At least I´m happy to sacrifice some elegance and cleanliness for efficiency and readability. That´s why I think before I write the first test, no, even before I prioritize test cases. And I think about the design of the code. What functional (!) structure should it have to make it easy to understand? How does that collide or play together with testability?

So I´m an architect while doing TDD. I´m a problem solver. And more often than not I´m not surprised by the coded solutions, but see them to comply with my ideas of how a solution should look like. I know the steps of converting arabic numbers to roman numbers before (!) I start coding. In fact I´m writing code to match my conceptual (algorithmic) approach.

Continuity in designing software on all levels

Architecture is thinking about structures, that is structures to deliver qualities. Solving the challenges of the problem domain ist about structures, that is structures to deliver functionality (in accordance with qualities).

Software is a self-similar system with many levels. It consists of wholes containing parts being wholes again for even smaller parts etc. “It´s holons all the way down” to use a term Arthur Koestler has coined. And because of that the same principles apply on all levels, not just DRY, SRP etc., but also “thinking helps”.

There is no discontinuity as Robert C. Martin suggest. Right to the contrary! Like there is strategic design, there is tactic design and situational design. And that means, there is thinking before doing. There is engineering before hammering and welding.

My interpretation of what Justin has written is: Use your head before you sling the code to make a test green. With that I agree wholeheartedly, even though I might disagree with some details of his approach.

The dichotomy Robert C. Martin suggests between architecture and problem domain does not warrant a fundamentally different approach to design on those different levels. Because design is about structure (and thus about options and decisions). And structure should not be an afterthought or be stumbled upon.

But before someone accuses me of advocating waterfall development: Yes, I do! :-) Because that´s how the world works. Before there is a man made structure, man better has thought about it. Design up-front is inevitable. Even if you want to you cannot avoid it. You just can hide it or deny it. But it´s there.

The question thus is not whether to design before coding or not. It´s not about waterfall or not. It´s just about the scope. If the scope encompasses so many requirements it would take you 3 months to design and 8 to implement, then that´s of course not gonna work. But if the scope is much, much smaller, say, 5 minutes of design and then 10 minutes of implementation… then that´s perfectly ok. The solution thus lies in ever smaller “waterfall iterations”. And – as always in life – it helps to be humble. If you devise a solution design don´t deem it above new insights. Always be prepared to revise it.

Which brings me back to Justin: I haven´t seen any TDD demonstration where 5 or 10 minutes of “design thinking” have been applied before coding. That´s sad. Because it makes it harder than necessary to learn to appreciate the power TDD can have. It´s a great tool – if not taken as the only one to come up with designs.


Thursday, October 3, 2013 #

Messaging - or programming with flows - is different from your grandpa´s object orientation. It´s so different, it needs time and effort to wrap your head around it. And it needs more information and perspectives than currently available.

There are some initiatives underway like NoFlo or the Reactive Manifesto plus all sorts of frameworks on different levels of abstraction. But what´s missing is down-to-earth and easy to understand literature for the average developer.

Paul Morrison´s Flow-Based Programming is a bit dated. Ted Faison´s Event-Based Programming is pretty platform specific. Gregor Hohpe´s Enterprise Integration Patterns is not easy to apply to every day programming tasks.

The other day, though, I stumbled across this Kickstarter campaign for a book on data flow programming from Matt Carkci. That sounds like what I´m looking for. A book explaining fundamentals written by someone with experience in current technologies; Matt seems a guy like you and me and pretty down to earth. I like that.

So I pledged for the campaign and would like to ask you to do it, too, if you´re interested in getting flow-design or messaging further off the ground.

In case you´re not yet convinced: Let Matt tell you in his own words, why even a single dollar would be a good investment:


Wednesday, September 11, 2013 #

I´ve seen many teams struggling with agility. Faithfully they were trying to live by Scrum or Kanban rules – but still it was hard to “get into the flow”. Day to day business seemed to interfere with the ideal of the process definition.

That made me think. What was missing? I´m a firm believer in the agile tenets of delivering incrementally, staying in close contact with the stakeholders for early/frequent feedback, and constant learning. So my guess is, if agility isn´t working yet, we need to try harder: think even more in increments, increase feedback, and learn harder.

The following articles sum up what I think this would lead to: We need to switch from thinking in terms of completing some software or feature to “just” constantly delivering small value increments. Feedback and progress are more important than anything else. Flow is more important than anything else. Which leads to decreasing batch sizes to the bare minimum: 1 day of work (i.e. 4-6 hours effort for a single person; more, if more people can reasonably be involved during that day).


Sunday, September 8, 2013 #

Today´s mainstream object orientation is not what Alan Kay, the inventor of the term, intended:

I invented the term object-oriented, and I can tell you that C++ wasn't what I had in mind.

To him, a core feature of object orientation was messaging, i.e. some form of “natural” communication:

I thought of objects being like biological cells and/or individual computers on a network, only able to communicate with messages.

Messaging was so important to him that he listed it as second in his definition of object orientation:

1. Everything is an object, 2. Objects communicate by sending and receiving messages [...]

And when asked, what´s lacking in today´s object orientation he explicitly said:

The big idea is “messaging”.

But what is messaging? How can we do it with todays mainstream OO languages?

That´s what I´ve thought about and wrote a couple of articles about in this blog. Here´s a chronological list of them for easy reference:

I think, if we really mean to do object oriented programming, we need to adopt messaging as a core feature of our code. I´d be glad if these articles help you to make the switch.