The Architect´s Napkin

.NET Software Architecture on the Back of a Napkin

  Home  |   Contact  |   Syndication    |   Login
  25 Posts | 0 Stories | 75 Comments | 0 Trackbacks

News

Twitter












Archives

Post Categories

Friday, December 23, 2011 #

Spinning as described in my previous article is all about flow. Its premise is: flow can emerge when work is partitioned in small, evenly sized chunks processed in a smooth manner.

There is a constant input of requests to the development team. A backlog is filled with strategically important requirements, support is reporting bugs, feedback requires changes, management wants to see ideas realized on short notice. Under these circumstances any plan becomes obsolete within a day or two. Or a lot of effort needs to be invested to stick to the plan.

Spinning is different. It does away with planning. So there is no plan anymore that can become obsolete. Instead each day the team decides what to accomplish, which means how to deliver a small value increment to the customer at the end of the day.

A value increment is…

  • …at least whatever a customer can give feedback on. In some form or another it demonstrates the team´s understanding of the requirements. This is of course done best with…
  • …working software the customer can Accept and actually use.

Value is generated whenever the customer´s trust is deepened. This happens when she feels to be understood. “Those guys really know what I mean! Yeah, they can relate to my problems. They home in on a solution. I can see it.” Working software is the best way to build trust. But sometimes less will do, too.

Software development is a communication process. That means messages have to go back and forth between customer and team. Spinning is setting a minimum frequency for this: every day the customer has to give feedback and tell the team what to do next (request), every day the team creates something valuable (response) to elicit feedback from the customer. Whatever is necessary to sustain this frequency should be done. Whatever stands in the way to obtain this frequency needs to be cut away.

Constantly moving closer to satisfaction is the purpose of Spinning. The basic means for that is to satisfy constantly – even if just a little bit. Satisfaction and trust are not built milestone by milestone or sprint by sprint, but day by day.

Whatever requests are fired at the team get processed day sized chunks:

image

The team is viewed as a “request processor” running in preemptive multitasking mode. There are several tasks to accomplish: a backlog needs to be burned down, bugs need fixing, operations need to be helped… A team not only needs to decrease latency, but also to hide latency.

With Spinning the ideal is to assign all developers to a single request per day (WIP=1). See the first three days in this picture:

image

On day 1 a request gets finished, on day 2 work starts at the next request. It continues on day 3.

But this ideal cannot be held up every single day. Sometimes developers need to attend to emergencies. If triage can schedule processing them for the next day, all´s well. Then developers just get allocated during daily work organization to more than one request.

image

But Spinning always reminds the team: the more developers are assigned to a single request, the better. More developers working on a request lead to faster implementation. Yes, producing a feature can be “industrialized” by distributing work. That´s at the core of Spinning; that´s why planning is so important before implementation. It´s all about speed to get feedback on working software.

On the other hand is illusionary to assume a team can focus on a single feature for several days. Support, operations, the boss´whims always cause disruption. Therefore the focus horizon is just one day. Most sudden requests can be postponed until the next day to be sliced, analyzed, organized, and implemented in an orderly way.

Sometimes however requests really require immediate attention. Such interruption of scheduled request increment implementation needs to be avoided. Hence it might be necessary to allocate one or two team members to be on standy-by for such work. If nothing happens they can work on a learning project or maybe a non time critical spike solution. But if an emergency calls for attendance, they are there – and the rest of the team can still quitely progress until the end of the day.

image

What´s the difference between Spinning and most teams? Most teams work like this, don´t they? They try to find a balance between progress and emergency handling. Yes, true. But they do so less systematically. And they constantly feel bad about it. They are working under the impression, software development should be different. They should be able to focus more. They should be able to keep the boss out of the team room for a couple of days. They should not be available for support.

Well, true, that would be much better – but unfortunately that´s not what the current culture of most projects allows. Spinning tries to take the burden of feeling bad off the backs of developers. It makes interruption the norm. Because every day the team is starting with a fresh view on what´s in the request queue.

Sure, teams doing Scrum and Kanban can organize themselves in a way to follow a process, and at the same time cater to emergencies. And they do. Every day. But this is outside of the process. The process is about focus. (Scrum more than Kanban.) To deal with different levels of urgency the process has to be tweaked.

Not so with Spinning. Spinning embraces imponderability. It´s defining a basic process mode

  • Focus on delivering customer value at the end of every day
  • Understand, plan, implement, check software in a systematic way in small increments
  • Understand, plan, check software as a team to reap the collective intelligence of the team members, and disemminate information in a natural way
  • Keep the Work-in-Progress at 1 – which can be different 1 each day

image

This helps to achieve two seemingly opposing goals:

  • React to changing priorities: Spinning satisfies the needs of different stakeholders (customer, support, manager) by daily allocating developers to requests according to what seems to be a valuable increment.
  • High speed development: Spinning satisfies the need for quick feedback by producing value every day with as many developers assigned to as few requests as possible.

Think of Spinning as a software production machine taking in raw material of very different size, shredding it into thin slices, and then transforming these slices into a steady flow of small increments of working software.

 

image

Spinning is true to the values of the Agile manifesto:

  • Individuals and interactions are highly valued. That´s why the team is supposed to work on understanding, planning, and checking collaboratively. This of course includes the customer. Also on the next level of abstraction the very frequency with which to elicit feedback from the customer during Acceptance is a testimony to this. Delivering working software and receiving feedback are interactions.
  • Working software is at the core of Spinning. Every day a working software increment has to be delivered. That´s also why Acceptance is so important to Spinning. When working on small increments on short notice, documentation loses value. Even a backlog loses value.
  • Customer collaboration is the very reason for Spinning. Acceptance by the customer is the most important activity during software development. Only in order to satisfy the customers strict Acceptance rhythm specifications need to be given. Specification follows from Acceptance, not the other way around. Also the high frequency of Spinning make contracts with concrete scope definitions and delivery plans obsolete, since the customer is supposed to steer the project on a daily basis.
  • Responding to change is the second reason for daily completion of requirement slices. No work in progress should be on a shelf at the end of the day. That ways the team is open to respond to any change the next day.

Through an orchestrated effort of individuals Spinning creates a constant flow of working software in close collaboration with the customer by responding to change at any time.

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Thursday, December 22, 2011 #

Agility needs to get onto the next level – that´s what I tried to explain in my previous articles. After a reality check – what´s missing from Agile practice? –, and some general musings about how a next level of Agility could look like, here now some very tangible suggestions.

Crank up the frequency

Current Agile practice is suffering from too little attention to Acceptance. To change this, very, very clear Acceptance dates need to be set. Acceptance can only get into a real pulling mode, if dates are fixed.

However, a date in weeks or months from now has no real pulling force on a team. This is obvious whenever you see a team change its working mode during an iteration because iteration end/delivery date is coming closer.

Acceptance always has to be around the corner. That means Acceptance has to be each and every day. That´s a frequency exerting real pull power over a team.

Yes, I mean it: A team should deliver a value increment every evening.

That´s how fast the development process should rotate at the core, at the implementation level. Delivering every day should be the default, the norm, the ideal. After sprinting comes Spinning.

Every evening (or next morning) the Acceptor should stand in the door of the team room an ask, “What have you got for me today?”

Again: I mean it. Requirements can and should be so finely sliced to be able to deliver value on a daily basis. This is possible, but it might require some training and good will. So please suspend your skepticism; don´t think, “This is impossible!” – but ask yourself “What could the benefits be?” Because there are many benefits to be reaped:

  • Benefit #1: The team gets feedback on a daily basis. What got implemented yesterday is checked today by the Acceptor – and immediately returned upon dissatisfaction. All mental state about what the feedback is about is still present in the team.
  • Benefit #2: If feedback is immediate and only a small increment has been added, not much waste can have been produced. Daily Acceptance transforms a project from ballistic flight to steered flight.
  • Benefit #3: Where Acceptance is daily, the customer can change his mind daily. The team is never caught in limbo. At the end of the day it´s done. So it can take up whatever challenge the customer proposes for the next day. The customer is most flexible in how he wants the software to evolve. Delivering on a daily basis for Acceptance make it perfectly ok for the customer to change his mind on a daily basis.
  • Benefit #4: The customer can stop work on features at any time. No features needs to be completed according to spec. This can safe quite some money, if a “good enough” state is reached before 100%. This can be checked for with daily Acceptance.
  • Benefit #5: Not only the customer can change his mind daily, also the team can. It can allocate its resources very flexibly every day. That way, even most emergencies need not interrupt the team; they just get postponed a couple of hours until the next “planning point”, which is not father away than tomorrow.

Yesterday´s waterfall software production was like shooting a movie. The script needed to fixed for production planning and shooting.

Today´s agile software production is like shooting a TV series. An overall plot needs to be pretty much fixed before production can start, but the script for each episode can be written later. Each episode is like a sprint, a season is like a milestone release.

Tomorrow´s elastic software production then looks like improvisational theatre. There is just a rough idea of what to present on stage. All else is decided on a moments notice. The piece evolves according to the interaction between the actors and the feedback from the audience.

Only cranking up the Acceptance frequency will lead to this kind of fluidity of development. And it´s the emphasis on Acceptance, which makes the difference. A daily build has been suggested since long. But a daily build does not equal daily Acceptance. It´s the daily involvement of the customer that counts. Daily Acceptance is about money. Because the money needs to be satisfied, and only money can exert any power on the software production process.

That´s why other benefits will only be realized once Acceptance is daily:

  • Benefit #6: The team will only change toward the better if Acceptance looms over it on da daily basis. Change will only happen, if it can demonstrate a connection to the money. Such connection is ever more easy to see, the closer the customer is. He´s most close if Acceptance is daily.
    Should a team do pair programming, reviews, TDD, should it refactor? Only if a positive impact on its performance can be shown the answer is yes. Performance, though, is under most tight de facto scrutiny when Acceptance is daily. Only changes to improve the capability to deliver at the end of the day true value are worthwhile to undertake.
  • Benefit #7: The code will only change toward better inner quality (aka evolvability), if it’s under constant pressure. Daily Acceptance puts it to the test daily. Every day unforeseen changes can be chosen for delivery in the evening. A code base not highly evolvable will break quickly under this kind of stress. That´s a feature of Spinning, not a drawback. It´s better to see code break in a couple of weeks or months instead of years. Fail fast is true for evolvability, too. If the code base is still small, it´s easier to change design/implementation habits that lead to its breakdown.

The Elastic development process

Spinning very specifically is about software development. Its framework knows about the stuff that gets produced. This is different from Scrum or Kanban, which can be applied to all sorts of efforts. Both are product or even industry agnostic. Spinning is not.

Spinning is ambitious. Daily Acceptance is easy to accomplish feat. Therefore Spinning needs to help as much as possible to reach this goal. Here´s the basic Spinning process. It´s a production process transforming requests into working software.

image

Every day the process is progressing, every day it rolls value out the door. Progress is achieved by two activities: develop and accept. First working software is developed, and then it´s accepted; and so on…

image

Development itself is an activity to “run in circles”, to be done iteratively over the course of a progress cycle: at least once, but maybe more often, depending on the request situation.

image

From the perspective of Spinning development is not a black box. Spinning has an opinion about how software development should be done:

  1. At the beginning of development the team needs to understand the problem. Too often developers jump onto implementation before they fully grasped what is supposed to be implemented in the first place. This results in a contra productive mixture of problem solving and coding. Also problem solving that way is often done alone; the intelligence of the whole team is not leveraged.
  2. After understanding comes planning. Planning is needed to make software development a truly collaborative effort. Without planning software development deteriorates to a group activity instead of a team activity. During planning the team develops its idea of a solution to the just analyzed problem. This greatly helps collective project ownership.
  3. It´s only after planning that implementation can start. During implementation developers individually work on code to contribute their share to the overall progress.
  4. Finally the implementation needs to be checked before it can be handed off to Acceptance.

Each of these fundamental steps again consists of sub-steps. Some of them are established and self-explanatory, some need clarification.

What understanding means, seems to be obvious. But in fact it is not. Teams often brush over understanding a request in order to get to coding more quickly. This can be seen in many Coding Dojos, too. Even the simplest Code Katas are not really understood before someone fires up an IDE and starts hacking away on a test; this leads to more refactoring than needed.

Spinning tries to slow down teams in this regard. High overall velocity can only be achieved if a team has a solid grasp of the problem.


image
 
  • Understanding a request requires first to analyze it: what´s the goal, what´s given, which resources are needed, what technologies could be helpful, have similar problems been solved already etc.? These and other questions need to be asked. By analyzing the request usually also a first approach to solving the problem is found.
  • To solve the problem and move it downstream towards implementation – keep the delivery at the end of the day in the back of your head! –, the request is sliced. It´s decomposed in sub-requirements until the team identifies at least on slice it is confident to be able to implement until the end of the day; or it has identified enough uncertainty to warrant further learning through a spike solution.
  • Triaging is needed to select the request(s) or request slice(s) to tackle until the end of the day. Each day the priorities can change for the customer or other stakeholders.

The order in which to analyze, triage, slice is not that important. Sometimes triage needs to be done first to zoom in on a request to actually analyze. Sometimes slicing comes right after analyzing to produce chunks that can be triaged. Sometimes the activities are repeated due to a deepened understanding of the problem.

In any case, only after requirements have been analyzed, sliced and triaged should planning start. Yes, planning. Spinning needs planning. Think of the proverb: “If you are in a hurry, go slower.” Planning is not a waste of development time, but actually a prerequisite for high velocity. Also as should be obvious from Spinning´s high frequency: planning is not “big plan upfront”. It´s “plan a little, implement a little”.

image

Planning is done in three different modes. And of course planning involves the whole team.

  • During planning the software architecture is designed. This means the team defines a structure for the software in non-functional terms. Of course this pertains only to the request(s) currently under development.
  • Architecture is only a framework for functionality. Hence planning continues with designing the functional structure for the current request slice(s). That´s what the model is about.
  • Finally, once the solution has been designed (in a coarse grained manner) it needs to be decomposed and assigned to the team members for concurrent implementation. Spinning means, all (available) team members are working on the same feature. Spinning favors the feature team over the feature developer; the goal is Work-in-Progress=1.
    That way collective code ownership is created in a natural manner. Daily standups become obsolete, because understanding and planning are done collectively, and implementation is planned and executed in a way so each developer knows her part as well as the larger context. Information is disseminated constantly during Spinning.

Whereas the team as a whole understands and plans, implementation is an individual task. Whoever wants to pair program is welcome, though. Compared to the other process steps, though, implementation is a single developer effort. That way the developers can reduce the latency of coding a feature by working in parallel.

image

Implementation consists of coding and testing. They are the responsibilities of the developers. No code without automated tests. Implementation manifests the architecture and the model in code. Since design and implementation take place on a daily basis, it does not matter much, if “bubbles don´t crash”. Design´s “bubbles” get translated so quickly into code, it´s almost as if developers were design diagram compilers.

Coding is led by explicit design. But coding of course is also designing – but on a more fine grained level than architecture and model. So if a developer wants to use TDD during coding, that´s perfectly ok. However the importance of TDD is diminished in Spinning (compared to XP), since it´s not the sole tool for designing software. Spinning favors explicit design before coding to underline the importance of a big picture mental model, and to support a collective understanding of the design.

The final development step is checking the code quality. Implementation produces a software increment to provide value to the customer. That´s at least the intention of understanding, planning, and implementing a request slice. But does it succeed?

image

  • A continuous build process first formally checks the quality of the result of the previous development steps. Does the code compile, do all tests run flawlessly, can it be deployed to different target platforms?
  • As important a continuous build is, it´s not enough. A code review is needed for several reasons. Firstly, only a code review can check if the design has been faithfully implemented. If deviations are not spotted and discussed immediately, the quality of the software structure quickly deteriorated and no common mental model exists anymore. Secondly, a code review is highly effective in spotting errors and deviations from requirements. Thirdly, during code review the whole team is back together to deepen its collective project ownership and share knowledge. Fourthly, during a code review any conventions can be checked.
  • Finally, after the build process and the developers have checked the code, quality can be assured by looking at the software from the outside. Some QA surely will be part of the automatic build process, but mostly there will be some work left for humans.
    To assure the team of the quality of its increment is an ongoing effort, although it looks like it´s just a process step at the end. Remember: understanding, planning, implementing, checking is done every day. And since people from QA belong to the team, they are involved from beginning to end.

In Spinning there is no “throwing code over the fence” like there is no “throwing a design over the fence”. A day in Spinning is mostly filled with the team members actually talking with each other by collectively understanding, planning, and checking the software.


image
 

Spinning only works if a group of developers really is a team – which is not the same. Or when there is no real team yet, Spinning helps to knit together the group members into a team. They almost can´t help but become a team, because otherwise the pull by daily Acceptance cannot be handled.

Summary

Spinning is serious about a couple of things:

  1. Software development is a true team effort. A cross-functional team is needed.
  2. Software development needs an explicit process to ensure high quality and constant flow. And this process needs to be specific to software development.
  3. Software development is only as good as Acceptance is.

Feedback and flexibility are increased, and running development in a tight circle minimizes detrimental effects of explicit planning and designing while increasing code quality and speed.

image

How exactly you do your request analysis, model before coding, organize the implementation or do your reviews… that´s up to you. If you find UML, TDD, pair programming helpful, that´s fine. You´ll soon notice if these tools are useful to keep up with the high frequency of Spinning.

Also Spinning is agnostic to whatever you think you need to wrap around it. You can use Spinning within Scrum to iteratively home in on your sprint commitment. You can use Spinning within Kanban to decrease batch size. Call Spinning an Agile core practice, if you like. In the end, though… Spinning alone should do the job.

View the process steps as items on a checklist. Check them off each day. By doing so ask the team: “What have we done regarding X?” with X being analysis, triage, …, review, QA. Make this your daily reflection habit.

Sometimes you´ll find you do more understanding or modeling than is necessary for today´s goal: deliver working software at the end of the day. That´s fine. Keep the insights in the back of your collective head for tomorrow. Just be sure to progress every day. Sometimes your steps will be bigger, sometimes they´ll be smaller. It´s not the size of your progress steps that matters much, but to continually progress. Deliver value to the customer each and every day. Call some request slice done each and every day. That will make your customer happy; it builds trust; and you can go home relaxed each night. No pending issues, no unfinished work. Spinning thus not only tries to be very true to the Agile manifesto, but also to the Cult of Done manifesto.

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Friday, December 16, 2011 #

In my previous article I came to a couple of conclusions based on the reality of software development, or should I say “the nature of software development”? Here are the – to me - undeniable facts of what our industry is all about:

  • Customers hardly know, what they want. Any specification is inherently fuzzy and incomplete. What fits the customer´s needs can only be determined by actually trying it out. The customer can only recognize a running piece of software as acceptable.
  • Because customers hardly know, what they want, they are likely to change their mind at any time. Due to unforeseeable circumstances requirements get changed or re-prioritized. On top of that software needs to be corrected, since we can´t help but deliver it imperfect. There will always be bugs and other unintended deviations from what the customer wanted to express with a requirements definition.

From this follows, I think:

  1. The most important role in a software project is the Acceptor. The Acceptor should ask for working software as often as possible. The Acceptor is the driving, no, pulling force behind a software endeavor. The Acceptor´s interest is to keep the software as close as possible on target. Any deviation from a straight line to the goal of a fitting software is costing money. In Lean speak deviations are waste.
    Since the goal is not clearly known – see above – the frequency with which to check working software is crucial. The higher the better.
    (As a side note: It should be obvious, that to ask for an estimation on when requirements will be implement is absurd for several reasons. Firstly, such an estimation will always be just rough; we´re in the business of development, not reproduction. Secondly, it´s an estimation for something fuzzy and incomplete. Thirdly, until the date is reached an unknown amount of waste will have been produced and there will be no chance of changing one´s mind. But I digress…)
  2. A high frequency of checking for acceptability leads to a high rate of change requests. Yesterday’s plan will be today’s rubbish. In addition reports from support sooner or later will call for unplanned changes in the course of software development. The second most important property of a software development process thus is high responsiveness. It has to embrace ad hoc requests. Somehow the seemingly contradictory has to be accomplished: smooth flow with regard to strategic goals and constant change.

Scrum and XP come from the world of the waterfall. They were born from frustration with the waterfall. It´s only natural they still contain traces from their predecessor process. This trace is the long iteration. It might not be long compared to waterfall milestones measured in months or years. But it´s still long with regard to how much can go wrong during the several weeks of an iteration.

Kanban comes from machine production. Machines and their parts are copies; they are unchanging manifestations of an already existing design. The variability in a reproduction business – although it might be high – hardly is as high as the variability in software development. Especially when interruptions are taken into account.

XP, Scrum, and Kanban have their origin outside/before software development. Maybe we should try to come up with a process from within software development reality? How would that look like?


A new set of priorities

A truly software oriented process would need to embrace the idiosyncracies of software development. Let me phrase that in the manifesto manner we´ve come to love. It´s my “Elastic Manifesto” for today:

•    Acceptance over specification
•    Progress over completion
•    Reactivity over commitment

Customers are interested in just one thing: working software, which means: value generating software. So whatever helps to satisfy this interested, we should do.

Remember: We´re talking about software here. This stuff is supposed to be most flexible, limber. So whatever mankind has learned with regard to much, much harder, inflexible, rigid stuff has to be taken with a grain of salt. Therefore we should start at the other end. We should start with what inevitably makes software development special.


Acceptance over specification

Whatever is just specified, whatever is just implemented is vain – until the customer accepts it. Only when money flows due to acceptance is software development successful. Only then a software requirement can be considered done.

With all the talk about pull-based software development we´ve to acknowledge, that true pull only exists if the customer is pulling first. So where is this true pull in Scrum, where is it in Kanban? I´m unable to see it, sorry.

Kanban is agnostic about acceptance. It just states, how a process should be organized at all. Look at various Kanban boards on the web and you´ll hardly ever see “Acceptance” as the final column. You´ll find QA, sure. But QA is not the customer. QA does not release any money even if they deem a piece of software without a flaw.

And then: Where´s the pull in Kanban anyway? No kanban is passed upstream. Kanban is more about suction than about pull. If there is a free place in a queue, it sucks in a done element from the previous process step. But this item has not been produced on demand. There is no explicit demand in Kanban; it´s implicit.

Putting Acceptance over specification is about producing explicit and very tangible demand, though. A specification of 500 pages is no demand. It´s just a document sitting on a shelf. Demand is, when a manager rushes into the team room questioning the team why the feature he ordered yesterday to please his golfing buddy isn´t ready yet.

The bitter truth is: Any change to how a team develops software will be difficult unless it´s positively tied to money. A software development team is not serving quality, it´s serving money. And money is only released upon acceptance.

Peter Drucker said: “Culture eats strategy for breakfast.” So if there is no explicit and very tangible culture of acceptance it´s very hard to implement any change not tied to money. That specifically pertains to changes costing money but not obviously generating more of it.

Suggestions to reduce expenses are always welcome. Adding another developer to increase productivity or buying a tool to achieve the same is also welcome. But what about TDD or refactoring or time for learning? They require spending money, but it´s not obvious whether they will help generate/save any.

And even though I like Clean Code, Refactoring, TDD etc. I have to admit: they are just tools. They are only justified if they help to please the customer. But how to determine that without putting acceptance at the heart of the development process?

The literature focuses on how to write specifications and being available for clarifications during a sprint. Acceptance is just a “necessary evil” at the end of a sprint after several weeks of work on several features. Or it´s delegated to QA. But as I said: QA might do what´s called acceptance testing; but QA does not release any money. So we better call QA´s job “preliminary acceptance testing” or “acceptance preparation”.

So how can acceptance be put at the heart of the development process? How can an Acceptor be put before the software development cart? This is done by setting delivery dates first. A delivery date is even more important than the requirements definition. A delivery date should be present before any requirements are talked about. And it has to be a very tight deadline. The Acceptor needs to be a strict taskmaster. He has to exert a constant pull on the team. He has to constantly ask: “Where´s the next increment I can check?” Again: This is more important than having a specification. Why? Because in the end what´s really producing value for a customer is what gets delivered and accepted, not what got specified and delivered.

Also, only by acceptance are specifications vindicated as not being waste.

Most ailing software projects I´ve seen are lacking strict, constantly pulling Acceptor. Many stakeholders are eager to push requirements into the backlog. But this not matched by an eagerness to pull working software from the team to check, if these requirements are met. Ask any stakeholder telling you a user story when and how often she´s going to be available for acceptance testing. You´ll not be surprised to either hear, “Leave me alone until you´re done. You´re the expert, you know when the software is done.” or “This should be the job of QA. Why else are we talking about acceptance criteria?” In the end, though, the same person will be the first to complain about how the software delivered misses the mark. And this will be very likely weeks if not months from the user story conversation.

How can a team possibly improve under such circumstances? It´s very hard. Because whatever it does is not tied to feedback from the source of money. If stress rises, newly learned tricks would quickly be forgotten, because there is no link to the money.

Without an Acceptor constantly pulling at the team with his finger on the money-release-trigger, changes to a team´s behavior beg the question of relevance. But if the Acceptor can´t help but say, “Wow, you stand firm while I´m pulling at you. You constantly deliver value; you don´t budge when I give negative feedback; your reactions are prompt. Here´s the money you truly earned.” Then and only then changes are worthwhile.


Progress over completion

Currently software development is focused on completely implementing any specification. A user story (or use case) is detailed into features, and then features are fully implemented in one go if possible. Only if a feature is estimated to take longer than a couple of weeks it is further cut up to be delivered in parts. The ideal is, though, to finish a feature during a single iteration. Get it done, and then get on to the next feature.

This to me seems in contradiction to two undeniable facts: specifications are of notorious low quality and customers have constantly changing priorities.

The quality of a specification cannot be determined before Acceptance. The understanding of the specification by the team cannot be determined before Acceptance.

So why should a team strive for completion of something of unknown quality? Why should a team wait for feedback on its understanding of a specification until its fully implemented?

Please note the difference between talking to a customer for clarification during an iteration and Acceptance. A team does not earn money by asking questions; it gets only paid after Acceptance. Asking a customer questions is some form of “belated specification” or just a “specification refinement”; it does not change the basic nature of specifications as being a priori descriptions. Only working software, software trying to deliver value, something tangible a customer can give feedback about is the ultimate specification. That's what makes software development so special.

Neither a customer nor a team thus should seek completion. Because why should something of questionable value be completed at all? Much better than completion is Progress, constant, sustainable, valuable Progress.

The notion of Progress might lack a completion date. But instead it promises true value. Value added is part of the definition of Progress.

Also Progress allows the customer to change her mind at any time. Value exists only in the eye of the Acceptor. Whether it´s of higher value to refine a feature beyond the current specification or to switch to a different feature or to even stop work on a feature before it has been completed according to the current specification… that´s a decision a customer should be able to make at any time.

Think of all the money that could be saved if features were not completed but only implemented up to a “good enough” level. What good enough means cannot be determined a priori; only acceptance and usage will tell. But if features are always fully implemented before acceptance, there is no (official) opportunity to change direction or cut it off before.

In current agile development processes feature implementation is a ballistic flight. Features are fired off at iteration start and hopefully hit the target at the end of the iteration.

What software development should look like, though, is the flight of a Cruise Missile. Each and every feature should be constantly monitored for its progress. Checking every couple of weeks is not enough. Otherwise, if the priorities change, there is no process conformant way to steer the feature. Despite current iterative practice customer needs are deemed to be pretty immobile targets. This to me seems contrary to project reality and customer needs.

Feature development needs to be under constant scrutiny by Acceptance. This means switching from a focus on completion to a focus on “mere” Progress.

Progress means a constant dialog between development team and customer. The team is telling the customer how it understands her needs by delivering small increments of working software. The customer tells the team what she needs by giving feedback on small increments of working software. It´s all about information exchange according to Gregory Bateson, it´s about differences which make a difference.

Customer as well as development team will benefit from a switch from completion to Progress. Progress means less stress for the team, Progress means more flexibility for the customer, Progress means more ROI for the customer.

Completion then becomes a non-issue: a feature or a whole software project is complete once enough value has been created, when it´s good enough – or when the project has run out of time/money. But until then maximum value will have been generated.


Reactivity over commitment

From Acceptance pulling on the team follows a switch of focus on Progress. And from focusing on Progress follows the necessity to constantly react to changing priorities.

The customer´s current will for completion is matched by the teams commitment to actually complete. Therefore is does not make sense anymore to commit a team to completion once the customer favors Progress. Reactivity becomes more important than commitment.

Value can only progressively be produced if a team is able to smoothly react to all the different categories of requests fire at it. Already today customers value responsiveness higher than completion. Responsiveness immediately builds trust. Watching people interact with software freezing during an operation easily proves this: they quickly get impatient and doubt the software is still trying to produce value for them.

That´s why we love operating systems with preemptive multitasking. That´s why we use multi-threading to hide the latency of an operation.

However, current development processes favor teams to be left alone until they have fulfilled their commitment. A Scrum sprint must not be broken by switching priorities midway. The very notion of breaking a sprint is testimony to this attitude. And once a feature has been picked up in Kanban by a process step, this process step must not be interrupted for fear of increasing the work in progress (WIP).

A sudden change in the trajectory of development is not really part of the picture of Agility, even though one could think of such Reactivity to be core to the notion of agility.

Reactivity seems to be in stark contrast to what´s beneficial for good software. For years we´ve heard about the bad effects of human multi-tasking. For years we´ve been admonished to help developers focus.

But maybe that´s one of the reasons why Agility hasn´t been so easily adopted by all teams? Maybe its current variations are too far from reality, from what´s necessary to attract money? Customers don´t care about focused developers. They care about value. And value to them does not only mean completed features but also catering to their whims.

Of course teams need to strike a balance between Reactivity and focus. But my guess is, we can still improve on Reactivity while not going insane by constantly switching focus or letting code quality drop even further.

For a start just a simple change of mind is necessary: If someone comes in with a request, welcome him. Get the request triaged. Each and every request signifies how valuable the software is today for someone, and how it would be even more valuable tomorrow with the request implemented.

Only after you´re able to welcome every request and not feel flooded or irritated, find a way to increase the Reactivity of the team. It needs to get into a mode where is can naturally adapt to high request frequency variations and high request priority variations.


Summary

Yes, I guess we need to arrive at a software development process from the idiosyncrasies of software development. This means its form should follow the money, which comes from Acceptance, and the essential volatility of specifications as well as customer priorities.

We need a more elastic process to not break under the pressure of request frequency variations and priority changes.

Current agile processes are old school in so far as they just shortened former waterfall iterations. It´s now weeks instead of months or years. But how many weeks should an agile iteration last? Well, it depends – that´s the common answer. From one to four weeks to be more concrete.

This to me looks like growing a social hierarchy down from the root. Which is bad because higher levels in the hierarchy beg legitimation from their base.

Why not turn this around? Why not grow iterations from the base, from the smallest timebox of reasonable size? That´s what I´m going to talk about in my next article.

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Wednesday, December 14, 2011 #

Let´s get real about software development: It´s never going to be a quietly flowing river. Never. And that´s why the current approaches to software development like XP, Scrum, and Kanban will always cause pain.

Their basic assumption is you should be able to isolate a team for a while to work on features. Leave it alone during an iteration or a sprint to complete a set of features, or at least sit still until the current feature is done.

Certainly that´s what we all want as developers: being able to concentrate on creating value by implementing new features or at least improving existing ones. Until we´re done.

Project reality seems to differ, though. And it does not just differ because Scrum or Kanban haven´t been implemented faithfully. Project reality will always differ because customers and users don´t care about our longing for being left alone.

Get real about emergencies

Software development is not like some factory production where orders are worked on in a smoothly flowing manner until fulfilled. Software development is more like an emergency unit in a hospital. It´s constantly under fire from all sides.

Requests are coming in at all times. Resources are always limited. That means a software team has to use triage (http://en.wikipedia.org/wiki/Triage) to determine what to do. Three request categories seem appropriate:

1.    Requests that must be answered immediately.
2.    Requests that should be answered soon. If not right away, then tomorrow is still ok, but not much later.
3.    Requests that can be answered at no specific date. Tomorrow would be ok, also next week, maybe even next month.

Category 1 is about rare bugs causing heavy damage with customers right now or in the foreseeable future; show stoppers so to speak. They need to be taken care of within a minutes or hours. Whether certain requests justifiably are categorized as show stoppers or not, is of no importance. Someone high enough up the ladder thinks a request is in category 1, so the team needs to react immediately.

Naturally category 1 requests should be rare. More than once a week seems problematic to me. Not because such interruptions are to be avoided, but because such urgent requests either hint at very poor software quality or an undue relationship with the customer base.

Category 2 is about most bugs. They need to be cared for pretty quickly, but not immediately. Bugs are a threat to any customer relationship. Customers rightly assume bug free software – with bug meaning an unintended deviation from what a software developer was trying to deliver. A bug is the fault of the development team so it needs to take care of it as quickly as possible.

Category 3 is about new features, feature change requests, and corrections due to uncovered misunderstandings or deeper insight. No life is at stake if such a request is not processed immediately. It can be put on a backlog for prioritization.

Any project not just started is under fire with all requests from all three categories. In an early greenfield project, requests might just fall into category 3. But as soon as customers/users have put their hands on some (early) release, category 2 and 1 requests will start to come in.

That´s a fact no software development process should deny or neglect. Right to the contrary any software development process should revolve around this basic fact of software development reality.

My conclusion #1: We need a software development process, which institutionalizes request triage.

Get real about uncertainty

At the heart of Agile and Lean is the realization that requirements are notoriously poor. As much as we invest in requirements elicitation… the result still is fuzzy. I´d even say: Customers cannot specify what they want, they can only recognize if that, what software development releases to them, is what fits their needs.

Customers are poor at a priori definition, but extremely good at a posteriori judgment.

As software developers we thus live in a constant state of uncertainty. We just don´t know if what we´ve developed really is what gets accepted by the customer. Did the customer define his requirements correctly? Are they complete? Are the comprehensive? Does the customer know at all what´s necessary to solve her problem? Have we understood the requirements? Have we come up with a solution matching our understanding? Will the customer accept our view?

Because of this uncertainty Agility/Lean advocates iterative development with delivery of completed features. Whatever get´s released to the customer at the end of an iteration has value to him; he can give feedback, because some requirement has been implemented.

How long until such a release? Two to four weeks seem the norm. Sometimes it´s just one week, sometimes it just takes as long as it takes for a feature. Iterations usually are measured in weeks, not months, not days or hours.

This seems ok – until iteration timeboxes collide with project reality. Any request of category 1 or 2 is likely to cause a delay. Such requests naturally are a nuisance for any process requiring quite time to work on a plan laid out for a couple of weeks.

My conclusion #2: We need a development process, which embraces the undeniable reality of category 1 and 2 requests.

But not only are timeboxes of several weeks in contradiction with daily project life, they also still do not really honor the uncertainty of requirements. Think of it: A day 1 of an iteration you start working on a feature. You finish it by day 4. But the only gives feedback at the end of the iteration, maybe on day 14 or 21. That´s a long time since you checked off the feature in your head as done and got rid of your mental feature state. The feedback loop is quite long even in today´s agile processes. On average it takes at least half an iteration until you hear back from the customer.

Also, during an iteration several features are implemented. This means, whatever change request the customer might have at the end of the iteration, it can possibly affect all features under development within the same iteration. The longer an iteration the more likely waste is created.

My conclusion #3: We need a development process, which allows for almost arbitrarily quick feedback from customers.

And then think of it: If requirements are so uncertain, why should a team deliver whole features at all at the end of a multi-week iteration? Neither does the team know, if it interpreted the requirements correctly, nor is it clear to the customer, if she really needs all that which she found important many weeks or months ago during requirements elicitation.

If a process tries to deliver as much of a feature as possible in a multi-week timebox, there is hardly any chance for the customer to say, “Stop! That´s good enough.” Current processes implement requirements in a very coarse grained manner. Their ideal is completion, arriving at Done for a whole use case or feature. That seems like a relic from the old waterfall days, to me. It´s not very true to the fundamental realization of Agility: Requirements are notoriously uncertain; we need to deliver often.

My conclusion #4: We need a development process, which allows customers to stop development of features at almost any percentage of completion.

Get real about evolvability

Before Agility the biggest challenge for software development was technology. We struggled to fulfill the explicit functional and non-functional requirements of customers.

The next big challenge then was correctness. Also a non-functional requirement, but not often explicitly stated. Customers rightly assume software to be correct, i.e. bug free. There is no excuse for a crash or an incorrect calculation.

Since then, though, the world has moved on. There are tons of technologies and tools at our disposal. There are several systematic approaches to increase correctness of code from reviews, to automatic unit tests, integration tests, and acceptance tests.

So I daresay today´s biggest challenge for software development is different. It´s harder to solve, because to assess the quality of our solutions with regard to it, is more difficult.

What I see as the biggest current challenge is maintainability or evolvability. We need to do a much better job keeping our code flexible. Cost is rising too steeply over the lifetime of a code base. Implementing a feature in month 3 of a project might cost X dollars, implementing it in month 18, though, will cost much, much more.

There are several approaches to attack this problem like Clean Code, Agile Architecture, or just tried and true principles and practices from 60 years of software development.

Strangely, however, current software development processes do not address the challenge. Neither Scrum nor Kanban are interested in code evolvability. They might measure a decrease in productivity, but they have nothing to say about how to improve the situation.

XP might be different in that regard. But then… how effective can it be to have list of principles and practices for high evolvability, when evolvability is not put to the test?

Yes, I think that´s a common blind spot of all current development processes: they do not really put evolvability to the test.

And why should they? Isn´t evolvability a matter of code design instead of process? Indeed it is – but alas there is hardly a measure for the quality of code base evolvability. Forget about LOC per method or class, forget about Cyclomatic Complexity, forget about afferent coupling etc. Those are just numbers that must be interpreted. And this interpretation is highly subjective and relative. Let´s get real: We can´t look at any set of measurements today and positively say, “This code has high evolvability.”

Checking the functional quality of code, checking it´s correctness, checking its non-functional qualities… that´s all different and comparatively easy. Checking evolvability, though, is hard. In the end the proof is only in the pudding. If a code base is hard to evolve, then, well, its evolvability must be low.

To check the fulfillment of functional requirements, execute the program. To check non-functional requirements, execute the program. To check the evolvability, evolve the code.

Sure, design/code developed using XP, Scrum, Kanban is put through evolutions more often than in a waterfall process. But my perception is, that´s not enough. There are so many teams trying to improve evolvability by applying Clean Code principles etc. – but the results are still suboptimal. From that I draw my last conclusion:

My conclusion #5: We need a development process, which addresses today´s major challenge evolvability head on by putting much more pressure on a code base through shortened evolution cycles.

Summary

Current development processes are a big step forward compared to whatever was the norm before. They have done a lot of good to the state-of-the-art of software development. To me the major contributions are:

  • We now know that requirements are very uncertain. The problem to solve is always poorly understood by customers and developers alike. To arrive at a solution we need to interact frequently.
  • In order to home in on a solution, a development team needs to deliver artifacts to customers on which they can give feedback. Software thus is delivered in slices (vertical sections), not layers (horizontal sections).

This corresponds to the first two items on the Agile Manifesto: Individual and interactions, and Working software.

However, I find the current implementations of the realizations of the Agile Manifesto lacking. Without doubt, they are well mend. But their implementation in many projects I´ve visited is strangely at odds with reality.

Teams are suffering even more from interruptions, because they now are trying to follow an ideal of not being interrupted. Customers are much controlled in their expression of how they think a project should continue. And teams have a hard time to assess how their code scores on an evolvability scale.

So my overall conclusion is: We need a next iteration of software development processes. XP, Scrum, Kanban are not the end, but just a beginning.

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Tuesday, July 05, 2011 #

Ron Jeffries challenged me to show how Flow-Design and Event-Based Components can help software development. This is the problem he posed in the Software Craftsmanship discussion group:

Solve bowling scoring. Here is the specification. Note that this is a simpler
version than the one Bob Martin often uses. I'll take questions if you have any.

  Given a list of the rolls of a legal game of ten pin bowling,
  which you may assume are provided without error or omission,
  produce the total, final, score of the game.

  Examples:
    Twenty zeros produce zero.
    Twenty fours produce 80.
    Twelve tens produce 300.

Since I don´t know much about bowling I consulted the KataBowling description as a second source. I haven´t done the kata before and have not looked at any of the kata solutions on display on the internet. That´s fortunate for a fresh and unbiased start – but it´s also a pain, since I simply don´t like bowling. But, well, I don´t want to complain. Here we go…

Understanding the problem

I believe in thinking before coding. And the first thing to think about is the problem. Before I can start to code I need to understand the problem which also means I have at least an idea of how to solve it.

It´s too bad I can´t show you the process of actually gaining an understanding of the problem. But what I can show you is data. Here´s a sample bowling game score sheet:

image

This shows you how I interpret the rules I read. Not just as a test case, but with some explanations as to why the scores are calculated like that.

The input to the solution I´m supposed to write would look like follows for this game:

image

It´s just the number of pins knocked down with each roll.

The combination of this list and the total score (here: 131) describes an acceptance test. Ron also provided me with some acceptance tests (see above).

Please note: Acceptance tests given by the customer are not really enough. You yourself should reply back to your customer with a couple of acceptance test suggestions. That´s what my game score sheet is about. Because only by defining your own acceptance test cases and showing them to the customer you can be sure to demonstrate you understand the problem.

If you explain something to someone – e.g. basic mathematical operations like +, –, *, / – and ask her, “Do you understand?” and she nods, that means nothing. You can only know if somebody understands what you´re saying by getting him to explain to you, what she understood or ask her questions.

The above description/acceptance test case is my reply to Ron. Hope he agrees I understood the problem correctly.

For the moment let me assume I did. What´s next?

“User Interface”

After I made sure I understand a problem, I like to clarify how the customer (here: Ron) wants to interact with my solution. Software is about transforming input into output. So how is the input provided? How should the output be returned? Often this entails user interaction and you need to talk with your customer about some kind of user interface. In this case, though, no UI is needed; at least Ron has not mentioned any. So I assume he´ll be satisfied with some kind of API. (As is the case for most Coding Katas.)

The API I´d like to suggest is this (Warning: I´ll be using C# as my solution language):

public class BowlingGame
{
    public static int CalculateTotal(IEnumerable<int> rolls) {…}
}

Using it would look like this:

var total = BowlingGame.CalculateTotal(new[]{3, 4, 5, …, 10, 3, 7, 5});

Acceptance Test Code

With acceptance tests and a UI in hand, I set up acceptance test code. This makes sure I do not deliver a solution not meeting the minimum criteria agreed upon my the customer. Employing NUnit this can look like this:

[Test]
[TestCase("00000000000000000000", 0)]
[TestCase("44444444444444444444", 80)]
[TestCase("AAAAAAAAAAAA", 300)]
[TestCase("192837465555647382915", 154)] // all spares
[TestCase("000000000000000000195", 15)]  // focus: just one more roll after spare in 10th frame
[TestCase("000000000000000000a34", 17)]  // focus: two more rolls after strike in 10th frame
[TestCase("3451a45267354aa375", 131)]    // acceptance test case from blog entry
public void AcceptanceTest(string rolls, int expectedTotal)
{   
    var total = BowlingGame.CalculateTotal(String2Rolls(rolls));
    Assert.AreEqual(expectedTotal, total);
}

The first three test cases are Ron´s. Then follow additional ones I derived from my understanding of the rules.

As soon as my solution delivers correct results for the acceptance test input data I´m ready to ship.

If I´d continue with TDD to design the implementation, I´d start from here. But I don´t. I continue with some more thinking…

Modelling the solution I

Instead of sitting down and code I like to close in on a solution using a stepwise refinement process. Here´s my first step; I draw the solution on the highes level of abstraction possible. The whole solution is a single black box:

image

There is a list of integers flowing into the one method representing the solution; and a single integer is flowing out of it as the result. Seen from far away that´s how the solution works, how in fact any solution works: rolls are transformed into a total score, input is transformed into output by a single action.

Given an action I can decide if I want to switch to coding mode. If I feel the solution will be short and I feel comfortable writing it down, I´d switch. But if not I´d rather continue modelling with a graphic language. It´s so much easier to change a mental model of a solution if it´s just a diagram.

In this case I decide to continue modelling not only because this is the purpose of the exercise, but also because I already have a clear vision of a more detailed design. Understanding the problem lead me to distinguish three operations:

Calculating the total score consists of adding the pins in each frame, adding bonus points for spares and adding bonus points for strikes.

I don´t think these operations are too far fetched. I find them pretty obvious from reading the requirements. I´d consider me not understanding the problem if I had no idea of these aspects of a solution.

But since I´m very sure these operations contribute towards the solution, I use them to refine the all encompassing action:

image

Now there are three functional units to put “domain logic” in – and one at the top just wiring them together. This is a stratified design: an operation on a higher level of abstraction is assembled from operations on a lower level of abstraction. A very well known approach to build complicated stuff all over the world.

But wait: there is a list of integers entering the root functional unit, but a list of frames go into the first operation. This does not look right. Where are the frames coming from? And where is the total score calculated? Two more functional units are needed:

image

This looks better: consistent and complete. This is even not very technical; I could explain it to the customer to make clear what kind of solution I envision.

Should I start coding now? I guess that would be ok. All functional units seem to be quite small. And they represent crucial domain terminology.

The basic idea of this solution is to transform the initial list of rolls into a set of frames – then enrich these frames with a score – and finally sum all frame scores. Hm… this sounds good. I should go back to the model and make it mirror this straightforward explanation:

image

That´s jojo-modelling, I´d say :-) First top-down, then bottom-up.

Now I can explain the solution to my fellow developers on three different levels. It´s complete on each level, but each level is lacking detail. That´s on purpose. Modelling with Flow-Design is about abstraction; and abstraction is about hiding details. That´s why Flow-Design does not want to duplicate what programming languages do. It´s not flow-charts, it´s data-flow. It´s not imperative, it´s declarative. The transformation depicted will magically happen like transforming data from sectors on a hard disk into records in memory magically happens when using SQL.

And what about the data? Usual object orientation start with focusing on the data.

With Flow-Design data is not the focus even though it´s about data-flow. Sure, in the end the details of the data flowing needs to be specified. But usually data is not the problem; how data should be structured mostly is pretty obvious. That´s probably one reason why you mostly start by modelling data structures: you feel comfortable, you get something done.

In the end, though, data is not really the problem. Transformation is. As programmers we´re hired to implement transformations. And since a well known advice is to start work with higher risk tasks I´d argue it´s good advice to start with the transformations when programming. Either data structures are well known – or transformations drive data structures (like TDD is supposed to drive design).

To start programming by identifying data classes from nouns in a requirements document might even be premature optimization. So be cautious about it.

But, yes, I need to define Frame before I start coding. Here it is:

class Frame
{
    public int[] Rolls = new int[2];
    public int Score;
}

More´s not necessary, I´d say. It´s devoid of functionality. Why? To be honest: I don´t know which functionality I should attribute to Frame. What´s the responsibility of Frame except to hold data? Responsibilities I feel sure about are modelled in the above diagram. To me it feels very natural to not force them onto frame.

Modelling the solution II

Although the above model seems to be up to the task, I don´t want to keep it as a secret: there is another way to model the solution. Maybe you even thought of this alternative first. It´s replacing the enrichment action sequence with parallel actions:

image

This is not to suggest there will be running anything in parallel (although it could). It´s only to make clear the independence of the actions to sum pins and bonuses.

Also note how frames enter the summation actions – but integers are leaving them. The list of frames will not get enriched. It´s just input to be traversed to calculate the output.

To me that sounds even better than the first model.

Nevertheless I´ll implement the first one because it lends itself nicely to a translation into very plain C# code as you´ll see. The second model I´ll leave to my colleague Stefan Lieser who´s working with me on Flow-Design and Event-based Components. I´ll describe it later in this blog.

For now I hope you feel with me at least a little bit how easy it is to reason about different approaches when looking at a picture. Imagine juggling models only in your head? Or imagine sitting at your IDE and coding away using TDD. You´ll most certainly focus on just a single solution. TDD will drive you into one direction without showing you the alternatives. Or if alternatives show up you´d need to experiment with them in code. Sure, that would be executable experiments – but it would also be quite tedious to explore them. You simply cannot type as fast as you can draw or think. (At least I cannot.)

So although “bubbles don´t crash” and there is no guarantee that the solutions are comprehensive I prefer to model them explicitly like this first. It´s sufficiently coarse grained to be swift. And it´s sufficiently fine grained as to make coding easier.

Translating the model

Enough scribbled. On to some code.

How should I start coding the first model?

Well, anyway I like. I can start top-down or bottom-up. The model tells me which functional units to code. I don´t need to find them out through refactoring, I know which ones are needed – at least at a certain level of abstraction.

So I randomly pick Add_pins_in_frame to implement first. Here´s my test – yes, I´m coding test-first:

[Test]
public void Calc_basic_scores()
{
    var frames = new[] { new Frame { Rolls = new[] { 1, 2 } },
                            new Frame { Rolls = new[] { 3, 5 } } };
    frames = frames.Add_pins_in_frame().ToArray();
    Assert.AreEqual(new[] {3, 8 },
                    frames.Select(f => f.Score).ToArray());
}

I chose to implement the action as a C# extension method. As you´ll see this will make the code very readable:

internal static class BowlingGameExtensions_Scoring
{
    public static IEnumerable<Frame> Add_pins_in_frame(this IEnumerable<Frame> frames)
    {
        foreach (var f in frames)
        {
            f.Score = f.Rolls[0] + f.Rolls[1];
            yield return f;
        }
    }   
    …

This is easy enough, isn´t it? A small functional unit, readily understandable, with single responsibility.

The other operations look the same. I spare you listing them here. They are all independent of each other. So they are easy to test.

But what about the composit functional units like the root? I implement them too although they are hardly doing anything. Their only purpose is to “wire up” the actions they contain. Here´s the API function for which you saw the acceptance test above:

public class BowlingGame
{
    public static int CalculateTotal(IEnumerable<int> rolls)
    {
        return rolls.ToFrames()
                    .Enrich_frames_with_score()
                    .Calc_total();
    }
}

How easy to read is this for you? How close to the model is this?

Or here the other composite action:

internal static class BowlingGameExtensions_Scoring
{
    public static IEnumerable<Frame> Enrich_frames_with_score(this IEnumerable<Frame> frames)
    {
        return frames.Add_pins_in_frame()
                     .Add_bonus_for_spares()
                     .Add_bonus_for_strikes();
    }   
    …

This too faithfully represents the model. In fact I “mechanically” translated it from the model. Each functional unit in the model either becomes an operation and is fleshed out test-first. Or it becomes a composite just plugging together other functional units in the most legible and easy way possible. There´s never a control statement in implementations of composite functional units.

There is a clear separation of concerns between operations and composites. The former need creative implementation, the latter only need mechanical implementation. They can even be generated from the model as Stefan Lieser´s solution will show.

Intermediate conclusion

Solving Ron´s problem this way was very straightforward. The hardest part was “decoding” the requirements. But once I understood the bowling score rules it took me just a couple of minutes to come up with the models.

As said above, even the nicest model diagram is no guarantee to be correct/sufficient. Nevertheless it provides a lot of value:

  • I was able to talk about my mental model with my colleague.
  • We were able to weigh the different approaches against each other.
  • The model described the most important part of any solution: the transformation. Flow-Design models are about functionality. The put action first and structure second.
  • The model provided me with small, focused, independent functional units to implement. No refactoring was necessary.
  • The model sports a fundamental and important separation of concerns between “doing” (operations) and “coordination” (composites).
  • The model is easy to understand on different levels of abstraction “at a glance”. (Which still requires understanding the visual notation and the domain terminology. My grandma sure would not understand these models.)
  • The model is fully present in the code, i.e. each model element has a corresponding code artifact.

But this is only one possible way of translating Flow-Designs into code. It´s the easiest way, because no tooling is required and the result is readily understood. However the code lacks the capability to reproduce the model; code and model can go out of sync. Also it can become tedious to write “coordinating” code by hand.

That´s why the next article is going to show you a translation of model into Event-based Components.

PS: In case you think this solution is overengineered: Partly I agree. It´s a tiny problem that also could have easily be solved without an explicit design like this. But Ron posed this problem to challenge me, so I was required to use Flow-Design.

On the other hand you never really know. A problem might look small – but in the end, once you understand it, is not. Also this approach lead to code that´s easy to evolve since it´s already refactored from the outset. Functionality is obvious and communicatable. Reasoning about where to apply changes if necessary is easy.

TDD might have resulted in a similarly fine grained design. But that design never had been visualized so all reasoning would need to work on just code. At least I find that cumbersome.

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Saturday, June 25, 2011 #

Doing CodeKatas is all the rage lately. That´s great since widely accepted exercises are important to further the art. They provide a means of communication across platforms and allow to compare results which is part of any deliberate practice.

But CodeKatas suffer from their size. They are intentionally small, so they can be done again and again. Repetition helps to build habit and to dig deeper. Over time ever new nuances of the problem or one´s approach become visible.

On the other hand, though, their small size limits the methods, techniques, technologies that can be applied. To improve your TDD skills doing CodeKatas might be enough. But what about other skills? Developing on a software in a team, designing larger pieces of software, iteratively releasing software… all this and more is kinda hard to train using the tiny CodeKata problems.

That´s why I´d like to present here another kind of kata I call Application Kata (or just AppKata).

AppKatas are larger programming problems. They require the development of “whole” applications, i.e. not just one class or method, but bunches of classes accessible through a user interface. Also AppKata problems always are split into iterations. To get the most out of them, just look at the requirements of one iteration at a time. This way you´re closer to reality where requirements evolve in unexpected ways.

So if you´re looking for more of a challenge for your software development skills, check out these AppKatas – or invent your own.

AppKatas are platform independent like CodeKatas. Use whatever programming language and IDE you like. Also use whatever approach to software development you like. Just be sensitive to how easy it is to evolve your code across iterations. Reflect on what went well and what not. Compare your solutions with others. Or – for even more challenge – go for the “Coding Carousel” (see below).

CSV Viewer

An application to view CSV files. Sounds easy, but watch out! Requirements sometimes drastically change if the customer is happy with what you delivered.

Questionnaire

If you like GUI programming, this AppKata might be for you. It´s about an app to let people fill out questionnaires. Also this problem might be interestin for you, if you´re into DDD.

  • Iteration 1
  • Iteration 2 (to come)
  • Iteration 3 (to come)
  • Iteration 4 (to come)

Tic Tac Toe

For developers who like game programming. Although Tic Tac Toe is a trivial game, this AppKata poses some interesting infrastructure challenges. The GUI, however, stays simple; leave any 3D ambitions at home ;-)

  • Iteration 1
  • Iteration 2 (to come)
  • Iteration 3 (to come)
  • Iteration 4 (to come)
  • Iteration 5 (to come)

Coding Carousel

There are many ways you can do AppKatas. Work on them alone or in a team, pitch several devs against each other in an AppKata contest – or go around in a Coding Carousel.

For the Coding Carousel you need at least 3 dev teams (regardless of size). All teams work on the same iteration at the same time.

But here´s the trick: After each iteration the teams swap their code. Whatever they did for iteration n will be the basis for changes another team has to apply in iteration n+1.

The code is going around the teams like in a carousel.

I promise you, that´s gonna be fun! :-)

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Sunday, March 20, 2011 #

In my previous post I summarized the notation for Flow-Design (FD) diagrams. Now is the time to show you how to translate those diagrams into code. Hopefully you feel how different this is from UML. UML leaves you alone with your sequence diagram or component diagram or activity diagram. They leave it to you how to translate your elaborate design into code. Or maybe UML thinks it´s so easy no further explanations are needed? I don´t know. I just know that, as soon as people stop designing with UML and start coding, things end up to be very different from the design. And that´s bad. That degrades graphical designs to just time waste on paper (or some designer). I even believe that´s the reason why most programmers view textual source code as the only and single source of truth. Design and code usually do not match.

FD is trying to change that. It wants to make true design a first class method in every developers toolchest. For that the first prerequisite is to be able to easily translate any design into code. Mechanically, without thinking. Even a compiler could do it :-) (More of that in some other article.)

Translating to Methods

The first translation I want to show you is for small designs. When you start using FD you should translate your diagrams like this.

Functional units become methods. That´s it. An input-pin becomes a method parameter, an output-pin becomes a return value:

image

The above is a part. But a board can be translated likewise and calls the nested FUs in order:

image

In any case be sure to keep the board method clear of any and all business logic. It should not contain any control structures like if, switch, or a loop. Boards do just one thing: calling nested functional units in proper sequence.

What about multiple input-pins? Try to avoid them. Replace them with a join returning a tuple:

image

What about multiple output-pins? Try to avoid them. Or return a tuple. Or use out-parameters:

image

But as I said, this simple translation is for simple designs only.

Splits and joins are easily done with method translation:

image

All pretty straightforward, isn´t it.

But what about wires, named pins, entry points, explicit dependencies? I suggest you don´t use this kind of translation when your designs need these features.

Translating to methods is for small scale designs like you might do once you´re working on the implementation of a part of a larger design. Or maybe for a code kata you´re doing in your local coding dojo. Instead of doing TDD try doing FD and translate your design into methods. You´ll see that way it´s much easier to work collaboratively on designs, remember them more easily, keep them clean, and lessen the need for refactoring.

Translating to Events

Translating FD diagrams to methods does not scale well. To reap all the benefits of FD you should therefore translate designs to Event-Based Components (EBC) as they were called originally. However this term is now deprecated, since the translation does not produce components in the sense of binary units of code. Nevertheless events enter the stage to hook together functional units.

Functional units

The favored translation results in a class for every functional unit with a method for every input-pin, and an event for every output-pin:

image

The default names for input-pins and output-pins are Process() and Result. You might find that strange since you´re thinking of classes as “things” with many responsibilities. In FD, though, classes can be very small with just one responsibility. And this responsibility is an action.

It´s not unusual to have a functional unit called “Read lines from text file” which then is translated into a class; why then should the class have a methode called ReadLinesFromTextFile()? Process() is sufficient.

If a functional unit has more than a single input-/output-pin the methods/events surely must have different names which are denoted by pin names:

image

Boards and parts do not differ in their basic translation. Both become classes (or interfaces, if you like). However, parts you implement yourself in some creative way. They are the workhorses. They contain the domain logic. Boards on the other hand could be generated – or are implemented by you without much thinking. Leave aside any creativity when implementing boards.

Boards do not contain (much) code in the input-pin methods. What they are doing happens in the constructor. Their sole purpose is to connect the functional units nested within them. Thats why input-pins are just delegating the work to be done. And that´s why all nested FUs are injected into a board using ctor injection:

image

With FD dependencies are (primarily) used to express nesting, i.e. different levels of abstraction. So a board is dependent on its nested functional units.

Within a flow, however, functional units do not (!) depend on one another. FU A does not depend on FU B or vice versa. Also X is independent of any other FUs preceding or following it. That makes testing extremely easy (see below).

Wiring

Wires in their simplest form are just event handler assignments:

image

This also makes it easy to translate split/fork:

image

Board input-/output-pins are treated a bit differently, though. That´s because they are connected to the same kind of pins, input to input, output to output:

image

Join

A join unfortunately is not that simple. You should create a small standard part to accomplish the task – or you use the join class from the ebclang project at CodePlex. ebclang is an effort to provide tools to help with Flow-Design. And ebcpatterns is a sub-project collecting implementations of parts to solve recurring problems.

Here is how you´d translate a join using the Join<T0, T1> class from ebcpatterns:

image

Sets of data items

Events can of course fire any number of times. So there is no need to distinguish between one output packet for an input packet and several output packets. Take as an example a functional unit splitting text lines into words:

image

One word or many… that does not make a difference for the translation of the output-pin to event Action<string>.

But there is a difficulty for any receiver of the output packets. Which word is the last word in a line? If several lines are processed and if it makes a difference to which line a word belongs, then a receiver has no way of associating a word with a line. The only way would be to send a special End-of-Line word as the last word of every line.

Fortunately there is another way of designing this. Just make it clear that output packets consist of several entries:

image

The star after the typename signifies the packet to be a list of data items. And a list is most simply represented by an IEmumerable<>.

Please note: This also works if the list contains millions of entries. Just don´t create an array and pass it along, but use an iterator (yield return) instead.

Explicit dependencies

Explicit dependencies are translated in a very explicit way. Instead of injecting them into the ctor an interface is used:

image

This has to advantages:

  1. Injection does not interfere with any other part of the implementation. It does not force a ctor or an additional parameter to the ctor, it does not require a base class.
  2. Injection is independent of object creation; injection can take place at any time during start-up of a Flow-Oriented application.

Entry Point

The entry point attribute is likewise translated to the implementation of an interface:

image

Configurable

Also making a part configurable is translated to the implementation of an interface:

image

Testing

Testing of functional units is easy:

With board you just do integration tests. Check only if the wiring is correct. Every path through a flow needs to be tested only once. Once you start using tools to generate boards these integration tests are not needed anymore.

With parts do unit tests as usual. Note that you don´t need a mock framework for that anymore because there are no dependencies between parts. Just pay attention to how to check the output:

image

You need to assign any relevant output event handlers before you call an input-pin method of the part. And don´t do the assert in the event handler because the test would go green even if the handler is not called.

Hosting – Putting it all together

Hosting the code created by this translation usually follows a pattern. It runs through a couple of phases:

  1. Build: Create all instances of functional units
  2. Bind: Wire-up the FUs by connecting output- to input-pins
  3. Inject: Inject explicit dependencies on all FUs implementing IDependsOn<T>
  4. Configure: Pass the command line args to all parts implementing IConfigurable
  5. Run: Call the Run() method with the command line args on the sole part implementing IEntryPoint

Build and Bind are put in a sequence to distinguish them; in reality, though, they are intertwined since binding happends also in ctors of boards upon creation.

image

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Saturday, March 19, 2011 #

You want to avoid the pitfalls of object oriented design? Then this is the right place to start. Use Flow-Oriented Analysis (FOA) and –Design (FOD or just FD for Flow-Design) to understand a problem domain and design a software solution. Flow-Orientation as described here is related to Flow-Based Programming, Event-Based Programming, Business Process Modelling, and even Event-Driven Architectures.

But even though “thinking in flows” is not new, I found it helpful to deviate from those precursors for several reasons. Some aim at too big systems for the average programmer, some are concerned with only asynchronous processing, some are even not very much concerned with programming at all.

What I was looking for was a design method to help in software projects of any size, be they large or tiny, involing synchronous or asynchronous processing, being local or distributed, running on the web or on the desktop or on a smartphone. That´s why I took ideas from all of the above sources and some additional and came up with Event-Based Components which later got repositioned and renamed to Flow-Design.

In the meantime this has generated some discussion (in the German developer community) and several teams have started to work with Flow-Design. Also I´ve conducted quite some trainings using Flow-Orientation for design. The results are very promising. Developers find it much easier to design software using Flow-Orientation than OOAD-based object orientation.

Since Flow-Orientation is moving fast and is not covered completely by a single source like a book, demand has increased for at least an overview of the current state of its notation. This page is trying to answer this demand by briefly introducing/describing every notational element as well as their translation into C# source code. Take this as a cheat sheet to put next to your whiteboard when designing software.

However, please do not expect any explanation as to the reasons behind Flow-Design elements. Details on why Flow-Design at all and why in this specific way you´ll find in the literature covering the topic. Here´s a resource page on Flow-Design/Event-Based Components, if you´re able to read German.

Notation

Connected Functional Units

The basic element of any FOD are functional units (FU):

image

Think of FUs as some kind of software code block processing data. For the moment forget about classes, methods, “components”, assemblies or whatever. See a FU as an abstract piece of code. Software then consists of just collaborating FUs.

I´m using circles/ellipses to draw FUs. But if you like, use rectangles. Whatever suites your whiteboard needs best.

 

The purpose of FUs is to process input and produce output. FUs are transformational.

image

However, FUs are not called and do not call other FUs. There is no dependency between FUs. Data just flows into a FU (input) and out of it (output). From where and where to is of no concern to a FU.

 

This way FUs can be concatenated in arbitrary ways:

image

 

Each FU can accept input from many sources and produce output for many sinks:

image

 

Flows

Connected FUs form a flow with a start and an end. Data is entering a flow at a source, and it´s leaving it through a sink.

image

Think of sources and sinks as special FUs which conntect wires to the environment of a network of FUs.

 

Wiring Details

Data is flowing into/out of FUs through wires. This is to allude to electrical engineering which since long has been working with composable parts.

image

Wires are attached to FUs usings pins. They are the entry/exit points for the data flowing along the wires. Input-/output pins currently need not be drawn explicitly. This is to keep designing on a whiteboard simple and quick.

 

Data flowing is of some type, so wires have a type attached to them. And pins have names. If there is only one input pin and output pin on a FU, though, you don´t need to mention them. The default is Process for a single input pin, and Result for a single output pin. But you´re free to give even single pins different names.

image

 

There is a shortcut in use to address a certain pin on a destination FU:

image

 

The type of the wire is put in parantheses for two reasons. 1. This way a “no-type” wire can be easily denoted, 2. this is a natural way to describe tuples of data.

image

 

To describe how much data is flowing, a star can be put next to the wire type:

image

 

Nesting – Boards and Parts

If more than 5 to 10 FUs need to be put in a flow a FD starts to become hard to understand. To keep diagrams clutter free they can be nested. You can turn any FU into a flow:

image

This leads to Flow-Designs with different levels of abstraction. A in the above illustration is a high level functional unit, A.1 and A.2 are lower level functional units.

One of the purposes of Flow-Design is to be able to describe systems on different levels of abstraction and thus make it easier to understand them. Humans use abstraction/decomposition to get a grip on complexity. Flow-Design strives to support this and make levels of abstraction first class citizens for programming.

You can read the above illustration like this: Functional units A.1 and A.2 detail what A is supposed to do. The whole of A´s responsibility is decomposed into smaller responsibilities A.1 and A.2. FU A thus does not do anything itself anymore! All A is responsible for is actually accomplished by the collaboration between A.1 and A.2.

image

Since A now is not doing anything anymore except containing A.1 and A.2 functional units are devided into two categories: boards and parts.

Boards are just containing other functional units; their sole responsibility is to wire them up. A is a board. Boards thus depend on the functional units nested within them. This dependency is not of a functional nature, though. Boards are not dependent on services provided by nested functional units. They are just concerned with their interface to be able to plug them together.

Parts are the workhorses of flows. They contain the real domain logic. They actually transform input into output. However, they do not depend on other functional units.

Please note the usage of source and sink in boards. They correspond to input-pins and output-pins of the board.

 

Implicit Dependencies

Nesting functional units leads to a dependency tree. Boards depend on nested functional units, they are the inner nodes of the tree. Parts are independent, they are the leafs:

image

Even though dependencies are the bane of software development, Flow-Design does not usually draw these dependencies. They are implicitly created by visually nesting functional units. And they are harmless. Boards are so simple in their functionality, they are little affected by changes in functional units they are depending on.

But functional units are implicitly dependent on more than nested functional units. They are also dependent on the data types of the wires attached to them:

image

This is also natural and thus does not need to be made explicit. And it pertains mainly to parts being dependent. Since boards don´t do anything with regard to a problem domain, they don´t care much about data types. Their infrastructural purpose just needs types of input/output-pins to match.

 

Explicit Dependencies

You could say, Flow-Orientation is about tackling complexity at its root cause: that´s dependencies. “Natural” dependencies are depicted naturally, i.e. implicitly. And whereever possible dependencies are not even created. Functional units don´t know their collaborators within a flow. This is core to Flow-Orientation. That makes for high composability of functional units.

A part is as independent of other functional units as a motor is from the rest of the car. And a board is as dependend on nested functional units as a motor is on a spark plug or a crank shaft. With Flow-Design software development moves closer to how hardware is constructed.

Implicit dependencies are not enough, though. Sometimes explicit dependencies make designs easier – as counterintuitive this might sound. So FD notation needs a ways to denote explicit dependencies:

image

Data flows along wires. But data does not flow along dependency relations. Instead dependency relations represent service calls. Functional unit C is depending on/calling services on functional unit S. If you want to be more specific, name the services next to the dependency relation:

image

Although you should try to stay clear of explicit dependencies, they are fundamentally ok. See them as a way to add another dimension to a flow. Usually the functionality of the independent FU (“Customer repository” above) is orthogonal to the domain of the flow it is referenced by. If you like emphasize this by using different shapes for dependent and independent FUs like above.

Such dependencies can be used to link in resources like databases or shared in-memory state. FUs can not only produce output but also can have side effects.

A common pattern for using such explizit dependencies is to hook a GUI into a flow as the source and/or the sink of data:

image

Which can be shortened to:

image

Treat FUs others depend on as boards (with a special non-FD API the dependent part is connected to), but do not embed them in a flow in the diagram they are depended upon.

 

Attributes of Functional Units

Creation and usage of functional units can be modified with attributes. So far the following have shown to be helpful:

  • Singleton: FUs are by default multitons. FUs in the same of different flows with the same name refer to the same functionality, but to different instances. Think of functional units as objects that get instanciated anew whereever they appear in a design. Sometimes though it´s helpful to reuse the same instance of a functional unit; this is always due to valuable state it holds. Signify this by annotating the FU with a “(S)”.

image

  • Multiton: FUs on which others depend are singletons by default. This is, because they usually are introduced where shared state comes into play.
    If you want to change them to be a singletons mark them with a “(M)”.

image

  • Configurable: Some parts need to be configured before the can do they work in a flow. Annotate them with a “(C)” to have them initialized before any data items to be processed by them arrive. Do not assume any order in which FUs are configured. How such configuration is happening is an implementation detail.

image

  • Entry point: In each design there needs to be a single part where “it all starts”. That´s the entry point for all processing. It´s like Program.Main() in C# programs. Mark the entry point part with an “(E)”. Quite often this will be the GUI part. How the entry point is started is an implementation detail. Just consider it the first FU to start do its job.

image

 

Patterns / Standard Parts

If more than a single wire is attached to an output-pin that´s called a split (or fork). The same data is flowing on all of the wires.

image

Remember: Flow-Designs are synchronous by default. So a split does not mean data is processed in parallel afterwards. Processing still happens synchronously and thus one branch after another. Do not assume any specific order of the processing on the different branches after the split.

 

It is common to do a split and let only parts of the original data flow on through the branches. This effectively means a map is needed after a split. This map can be implicit or explicit.

image

 

Although FUs can have multiple input-pins it is preferrable in most cases to combine input data from different branches using an explicit join:

image

The default output of a join is a tuple of its input values. The default behavior of a join is to output a value whenever a new input is received. However, to produce its first output a join needs an input for all its input-pins. Other join behaviors can be:

  • reset all inputs after an output
  • only produce output if data arrives on certain input-pins

[Continue with part 2 of the cheat sheet series]

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Saturday, January 08, 2011 #

“Write great code and everything else becomes easier” is what Paul Pagel believes in. That´s his version of an adage by Brian Marick he cites: “treat code as an end, not just a means.” And he concludes: “My post-Agile world is software craftsmanship.”

I wonder, if that´s really the way to go. Will “simply” writing great code lead the software industry into the light? He´s alluding to the philosopher Kant who proposed, a human beings should never be treated as a means, but always as an end. But should we transfer this ethical statement into the world of software? I doubt it.

 

Reason #1: Human beings are categorially different from code. They are autonomous entities who need to find a way of living happily together. To Kant it seemed this goal could only be reached if nobody (ab)used a human being for his/her purposes. Because using a human being, i.e. treating it as a means, would contradict the fundamental autonomy and freedom of human beings. People should hold up a symmetric view of their relationships: Since nobody wants to be (ab)used, nobody should (ab)use anybody else. If you want to be treated decently, with respect, in accordance with your own free will - which means as an end - then do the same to other people.

Code is dead, it´s a product, it´s a tool for people to reach their goals. No company spends any money on code other than to save money or earn money in the long run. Code is not a puppy. Enterprises do not commission software development to just feel good in its company. Code is not a buddy. Code is a slave, if you will. A mechanical slave, a non-tangible robot. Code is a tool, is a tool. And if we start to treat it differently, if we elevate its status unduely… I guess that will contort our relationship in a contraproductive way.

Please get me right: Just because something is “just a tool”, “just a product” does not mean we should not be careful while designing, building, using it. Right to the contrary. We should be very careful when writing code – but not for the code´s sake! We should be careful because we respect our customers who are fellow human beings who should be treated as an end.

If we are careless, neglectful, ignorant when producing code on their behalf, then we´re using them. Being sloppy means you´re caring more for yourself that for your customer. You´re then treating the customer as a means to fulfill some of your own needs. That´s plain unethical behavior.

 

Reason #2: The focus should always be on your purpose, not on any tool. But if code is treated as an end, then the focus is on the code. That might sound right, because where else should be your focus as a software developer? But, well, I´d say, your focus should be on delivering value to your customer. Because in the end your customer does not care if you write a single line of code. She just wants her problem to be solved. Solving problems is the purpose of any contractor.

Code must be treated just as a means, a tool we know how to handle very well. But if we´re really trying to be craftsmen then we should be conscious about exactly that and act ethically. That means we must never be so focused on our tool as to be unable to suggest better solutions to the problems of our customers than code.

 

I´m all with Paul when he urges us to “Write great code”. Sure, if you need to write code, then by all means do so. Write the best code you can think of – and then try to improve it. Paul has all the best intentions when he signs Brians “treat code as an end” - but as we all know: “The road to hell is paved with best intentions” ;-)

Yes, I can imagine a “hell of code focus”. In fact, I don´t need to imagine it, I´m seeing it quite often. Because code hell is whereever two developers stand together and are so immersed in talking about all sorts of coding tricks, design patterns, code smells, technologies, platforms, tools that they lose sight of the big picture.

Talking about TDD or SOLID or refactoring is a sign of consciousness – relative to the “cowboy coders” view of the world. But from yet another point of view TDD, SOLID, and refactoring are just cures for ailments within a system. And I fear, if “Writing great code” is the only focus or the main focus of software development, then we as an industry lose the ability to see that.

Focus draws a line around something, it defines a horizon for perceptions and thinking. So if we focus on code our horizon ends where “the land of code” ends. I don´t think that should be our professional attitude.

 

So what about Software Craftsmanship as the next big thing after Agility?

I think Software Craftsmanship has an important message for all software developers and beyond. But to make it the successor of the Agility movement seems to miss a point.

Agility never claimed to solve all software development problems, I´d say. So to blame it for having missed out on certain aspects of it is wrong. If I had to summarize Agility in one word I´d say “Value”. Agility put value for the customer back in software development. Focus on delivering value early and often – that´s Agility´s mantra. All else follows from that.

And I ask you: Is that obsolete? Is delivering value not hip anymore? No, sure not. That´s our very purpose as software developers. So how can Agility become obsolete and need to be replaced?

We need to do away with this “either/or”-thinking. It´s either Agility or Lean or Software Craftsmanship or whatnot.

Instead we should start integrating concepts and movements. Think “both/and”. Think Agility plus Software Craftsmanship plus Lean plus whatnot. We don´t neet to tear down anything from a piedestal and replace it with a new idol. Instead we should do away with piedestals and arrange whatever is helpful is a circle. Then we can turn to concepts, movements for whatever they are best.

After 10 years of Agility we should be able to identify what it was good at – and keep that. Keep Agility around and add whatever Agility was lacking or never concerned with. Add whatever is at the core of Software Craftsmanship. Add whatever is at the core of Lean etc.

But don´t call out the age of Post-Agility. Because it better never will end. Because once we start to lose Agility´s core we´re losing focus of the customer.

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Sunday, August 08, 2010 #

Logging, validation, exception handling: that´s easy aspects to insert into an Event-Based Components design as I´ve shown in my previous post. But what about multi-threading? Or better: parallel and asynchronous processing?

In this article I want to show you, how you could approach multi-core programming using aspects you insert into an existing EBC architecture.

Asynchronous processing

imageWhy use multiple threads at all? It´s because you either want to hide latency, or you want to decrease latency, or you want to increase throughput. Hiding latency means you want some client not to wait for a service to finish before it can go on an do other stuff.

It´s like when you call the pizza delivery service: You order a pizza, then you hang up the phone, and do whatever you like until the pizza is delivered. However long it takes to bake the pizza – you need not freeze during that period. The latency of pizza baking is hidden from you. The pizza service runs asynchronously with regard to its customers.

How could hidden latency help the file indexing scenario? Well, it could make the whole process asynchronous. So whoever calls the indexer does not need to wait for it to finish. This might no be a pressing issue as long as the indexing is wrapped up in a console application. But what if I made the indexing functionality a library?

The Indexer.Console project references the Indexer library. The whole indexing functionality is hidden in in class IndexFiles:

image 

Now any client application can use the indexer like this:

var index = new IndexFiles();
index.Out_Statistics += stats =>
    {
        System.Console.WriteLine("Successfully indexed {0} words.", stats.WordCount);
    };

index.Out_ValidationError += err =>
    {
        System.Console.WriteLine("*** Aborted indexing! Validation error: {0}", err);
    };

index.Out_UnhandledException += ex =>
    {
        System.Console.WriteLine("*** Aborted indexing! Unexpected exception: {0}. See log for details.", ex.Message);
    };

index.In_Process(args[0], args[1]);

And sure a client application would not want to wait for the indexer to finish its work of indexing a whole hard disk with tens of thousands of file. So making the indexing process asynchronous would be a very nice thing to do. But how?

Well, just put an Asynchronize activity right at the start of the feature process and you´re done:

image

Any activities the Asynchronize activity passes data on to are run on a different thread than the one from which the data originated. However, all data passing through Asynchronize is processed on the same thread! That means the data is processed sequentially although in parallel to other code.

The Asynchronize activity creates a single thread and a queue. Data coming in is queued up and the thread is picking data items from the queue whenever it has become idle. If nothing´s left to do it will wait for a notification that new data has arrived.

public class Asynchronize<T>
{
    private readonly Thread worker;

    private readonly Queue<T> dataToProcess;
    private readonly AutoResetEvent dataAvailable;


    public Asynchronize()
    {
        this.dataToProcess = new Queue<T>();
        this.dataAvailable = new AutoResetEvent(false);

        this.worker = new Thread(DispatchDataSequentially) {IsBackground = true};
        this.worker.Start();
    }


    private void DispatchDataSequentially()
    {
        while (true)
        {
            this.dataAvailable.WaitOne();

            T data;
            while (TryGetDataToDispatch(out data))
            {
                this.Out_ProcessSequentially(data);
            }
        }
    }


    private bool TryGetDataToDispatch(out T data)
    {
        data = default(T);

        lock (this.dataToProcess)
        {
            if (this.dataToProcess.Count() == 0) return false;

            data = this.dataToProcess.Dequeue();
            return true;
        }
    }


    public void In_Process(T data)
    {
        lock (this.dataToProcess)
        {
            this.dataToProcess.Enqueue(data);
        }
        this.dataAvailable.Set();
    }


    public event Action<T> Out_ProcessSequentially;
}

This way the latency of processing the data is hidden from its source, but at the same time the order of data item processing it retained.

To plug this aspect in is as easy as plugging in the validation aspect or the exception handling aspect:

public IndexFiles()
{
    // Build
    ...
    var asyncCompileFiles = new Asynchronize<Tuple<string, string>>();
    ...

    // Bind
    this.in_Process = _ => asyncCompileFiles.In_Process(_);

    asyncCompileFiles.Out_ProcessSequentially += handleEx.In_Process;
    handleEx.Out_Process += compileFiles.In_Process;
...

The Asynchronize aspect activity passes on the data it receives – but does so on a single different thread. It´s a generic functional unit. Very simple. It does one job.

If you want more, like exception handling for background processing, then you can combine it with the Handle exception activity into a composite activity, e.g.

image

EBC activities lend themselves to aggreation. It´s very easy to compose them into activities of higher order. So feel free to start implementing your own pluggable application building blocks.

So now the whole indexing process is running on a background thread. Client code is free to work on other stuff and will receive notifications upon completion or exceptional situations.

To exploit a multi-core CPU, though, indexing should happen on more than one thread. By using the Asynchronize activity a second time this can easily be accomplished. With it building the index can be decoupled from Crawling the directory tree and Extracting words:

image

See how the stages of the feature process are running on different threads (marked with their own background color). That means they are working in parallel: the initiator can continue while the compiling and extracting is happening. The compiling and extracting can continue while the building is happening.

Please note: Build index is also working sequentially. So only 3 threads are in use while indexing: initiator thread, compile thread, index builder thread. This ensures maximum speed for both stages on a two core CPU.

In addition Build index does not have to take precautions against concurrent access to its resource, the index. Only a single thread is working on it. That would have been different if Asynchronize would have just used the ThreadPool to dispatch data to a background thread. Stateful asynchronous operations thus do not inevitably lead to headaches ;-)

Switching to a data flow protocol

In my first posting I mentioned two different possible ways of passing data through the feature process: using a data flow protocol or IEnumerable<>. So far IEnumerable<> was just a perfect choice. The index could be written as soon as the IEnumerable<> passed to Build index was exausted.

Now that Build index runs on its own thread, though, it´s not possible to pass an IEnumerable<> to it anymore. Each data item has to be delivered separately. Thus I need to switch to a data flow protocol between Extract words and Build index.

That´s makes Compile words look a little ugly:

public class Compile_words
{
    private Index index = new Index();


    public void In_Process(Tuple<string, string[]> input)
    {
        if(input != null)
        {
            Trace.TraceInformation("Compile words({0}, {1} words) [Thread {2}]", input.Item1, input.Item2.Length,
Thread.CurrentThread.GetHashCode());

            foreach (var word in input.Item2)
                this.index.Add(word, input.Item1);
        }
        else
        {
            this.Out_IndexCompiled(this.index);
            this.Out_Statistics(new IndexStats(this.index.WordCount));

            this.index = new Index();
        }
    }


    public event Action<Index> Out_IndexCompiled;
    public event Action<IndexStats> Out_Statistics;
}

Nevertheless it´s straightforward, I´d say. Just checking for the terminating null is strange.

Parallel processing

What if you´ve more than two cores available? What if you not only want to hide latency, but decrease latency? Then not only asynchronous, but parallel processing is your friend.

This can be as easily introduced into an EBC design. We could, for example, detach Extract words from Compile files so that all files are analyzed in parallel. A Parallelize activitiy is very simple to implement:

public class Parallelize<T>
{
    public void In_Process(T data)
    {
        ThreadPool.QueueUserWorkItem(_ => this.Out_ProcessInParallel((T)_), data);
    }

    public event Action<T> Out_ProcessInParallel;
}

ThreadPool is the only help you need from the .NET framework. Each data item is dispatched for processing on another thread (within the limits of the thread pool). Thus they are all handled in parallel.

Technically it´s easy to insert the Parallelize aspect activity into the EBC design: just plug it between the two activities.

image

But this is not sufficient. It wouldn´t work. Don´t do it just like that!

The problem with this is, it´s not guaranteed that the terminating null is the last data item output by Extract words. There are potentially many Extract word activities running at the same time. And any one of them could issue a null – even if other ones are still analysing a file.

image

To guarantee null is sent on to Build index only after all other data items have been processed by Extract word threads a trick needs to be employed: the feature process needs to count how many “tasks” were send to Extract words…

public class CountItemsUntilNullForScatter<T> where T : class
{
    private int counter = 0;


    public void In_Count(T item)
    {
        if (item == null)
        {
            Trace.TraceInformation("CountUntilNull({0}) [Thread {1}]", this.counter, Thread.CurrentThread.GetHashCode());

            this.Out_Count(this.counter);
            this.counter = 0;
        }
        else
        {
            this.counter++;
            this.Out_Counted(item);
        }
    }


    public event Action<T> Out_Counted;
    public event Action<int> Out_Count;
}

and then count how many results were output by Extract words. Then after as many results left the activity as tasks had entered it the null must be inserted into the data stream:

public class InsertNullAfterItemsForGather<T> where T : class
{
    private int numberOfItemsToGather;
    private int count;


    public void In_Process(T item)
    {
        this.Out_Gather(item);

        lock (this)
        {
            this.count++;
            IssueNull();
        }
    }


    public void In_NumberOfItemsToGather(int numberOfItemsToGather)
    {
        lock (this)
        {
            this.numberOfItemsToGather = numberOfItemsToGather;
            IssueNull();
        }
    }


    private void IssueNull()
    {
        if (this.count >= this.numberOfItemsToGather)
        {
            this.Out_Gather(null);
            this.count = 0;
        }
    }


    public event Action<T> Out_Gather;
}

This is a common pattern and is called scatter-gather. Parallelize scatters work to be done across a number of threads. And a downstream activity gathers the results for further processing – because that´s supposed to be done sequentially, not in parallel.

image

The feature process now works as follows:

  1. A single path is passed to Compile files and is processed asynchronously in some thread T0.
  2. Compile files outputs n filenames terminated by a null.
  3. Count data items counts the number of filenames and passes them on to Parallelize.
  4. Parallelize dispatches processing of those filenames by Extract words to threads T1..Tn (or possibly fewer depending on the thread pool size and the time it takes to process each file).
  5. Extract words is run n times in parallel and issues n tuples containing the words found in the files. The activity “swallows” the null filename, an no null results are output. This made the Extract words activities a bit simpler.
  6. Insert null counts the number of results flowing out of Extract words and inserts a null into the data stream after n items. n had been passed to it by Count data items once it determined how many filenames were generated.
  7. Sequentially processing of the words found is dispatched to thread Tn+1 by Asynchronize. Once Build index encounters the inserted null it write the index to a file.

If you look at the code of the activities you realize they are free of any multi-threading concerns. That makes them easy to test. Asynchronous/parallel processing is an orthogonal aspect and has been completely wrapped into a couple of standard activities which can be combined with domain activities.

However asynchronous processing does not come for free. Some changes needed to be made like moving from IEnumerable<> to a data flow protocol. But I´d argue they did not complicate the business logic.

Sure, async/parallel processing can be more complex. But my feeling is that EBC make it easier to deal with it. With EBC the async/parallel aspect is made visible and very explicit as the above diagrams attest. It´s clearly separate from any domain code. You can reason about the async/parallel processing by walking along graphical feature processes – which then are translated almost mechanically into code.

Also it´s obvious, where synchronizations between threads need to take place.

image

Each dot marks where two running threads meet. At these points you need to think about how to organize access to shared resources. Because data can be passed from one thread to another only via such resources. Which brings me to another isssue…

Synchronization with WinForms main thread

There is a special kind of resource to which not only access from different threads needs to be synchronized, but which can only be accessed from a special thread: most controls on WinForms forms. Access to them is not thread safe by default. So if data arrives on some other thread it needs to be transfered to the main WinForms thread before it can be displayed.

This is another aspect and can be solved by plugging an activity between an output and a WinForms input:

image

I´ve added another project to the indexer sample solution to implement a WinForms frontend:

image

The form is the initiator/data source of the feature process as well as its sink. It´s easy to wire it up to the IndexFiles process (which I´ve adorned with an output for each file found as a progress signal):

var dlg = new WinIndex();
var indexer = new IndexFiles();


dlg.Out_Index += indexer.In_Process;

indexer.Out_FileFoundToIndex += SwitchBackToSyncContext<string>.Wrap(dlg.In_FileFound);
indexer.Out_Statistics += SwitchBackToSyncContext<IndexStats>.Wrap(dlg.In_IndexStats);
indexer.Out_ValidationError += SwitchBackToSyncContext<string>.Wrap(dlg.In_ValidationError);
indexer.Out_UnhandledException += SwitchBackToSyncContext<Exception>.Wrap(dlg.In_Exception);

SwitchBackToSyncContext<>.Wrap creates an aspect activity to be inserted before the one passed to it.

public class SwitchBackToSyncContext<T>
{
    private readonly SynchronizationContext ctx = SynchronizationContext.Current;


    public SwitchBackToSyncContext() : this(SynchronizationContext.Current) { }
    internal SwitchBackToSyncContext(SynchronizationContext ctx)
    {
        this.ctx = ctx;
    }


    public void In_Process(T msg)
    {
        if (this.ctx != null)
            this.ctx.Send(this.Out_ContinueInSyncContext, null);
        else
            this.Out_ContinueInSyncContext(msg);
    }


    public event Action<T> Out_ContinueInSyncContext;


    public static Action<T> Wrap(Action<T> continuation)
    {
        var switchBack = new SwitchBackToSyncContext<T>();
        switchBack.Out_ContinueInSyncContext += continuation;
        return switchBack.In_Process;
    }
}

The aspect activity switches execution back to the synchronization context on which it was created (which is the same as the sync ctx of the WinIndex form).

Check out the code at Google Projects here. It´s the branch leading to the complete async solution including the WinForms frontend.

Summary

This concludes my introduction to Aspect-Oriented Programming (AOP) with Event-Based Components. I hope I was able to show you, how easy it is to insert aspects into EBC designs. But it´s not only easy to do technically. The resulting design is also easy to understand, because aspects are made explicit.

No special tooling is required and yet you´re very flexible. Quickly swap in/out aspects without burdening your domain logic.

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati