The Architect´s Napkin

.NET Software Architecture on the Back of a Napkin

  Home  |   Contact  |   Syndication    |   Login
  49 Posts | 0 Stories | 136 Comments | 0 Trackbacks

News

Archives

Post Categories

Steve Bate just has written two very hones and straightforward blog articles [1,2] on how he approaches programming. I very much agree with him. For some four year´s I´ve explored programming along the same lines – and have experienced much relief. Programming has become easier and more fun. Many of the problems I experienced before and others are complaining about have disappeared.

Just take Mock Frameworks as an example: I´m not using them anymore. There´s no need for them. I don´t need to stay informed on the latest Mock Framework development, I don´t to select any, I don´t need to install one, I don´t need to learn how to use them in dozens of different situations. That has freed up time I can now invest in other areas, which actually might earn me money. Because, let´s be honest, using Mock Frameworks are not what we´re getting paid for. They might be necessary evils, to make testing of certain structures easier. But using them is nothing to strive for.

Anyway… What I wanted to say is: I thank Steve for coming out with how he´s constructing his code. And I want to support him. By saying “Don´t be ashamed, I´m doing it too!” – and by adding some more fundamental thoughts to his pragmatic presentation.

Steve´s description is concrete. He shows code of how he´s living a message passing programmer´s life. But despite some general thoughts on what OOP and message passing could mean, he does not define it. But what exactly is message passing or messaging? For Alan Kay it was at the very heart of object orientation:

I thought of objects being like biological cells and/or individual computers on a network, only able to communicate with messages (so messaging came at the very beginning -- it took a while to see how to do messaging in a programming language efficiently enough to be useful. [3]

Unfortunately Alan Kay did not explain, what he actually meant by “messaging”, except for an analogy to biology. Sure, we could turn to Smalltalk and see how it´s done there. But that is just a concrete implementation, not a definition.

So what is messaging? Why can Steve call his programming style messaging? Why does he rightfully criticise mainstream OOP as not implementing messaging? (At least I think his criticism is valid.)

I´d like to try to answer this with my own 2c definition of messaging. It´s very, very simple:

Messaging is one-way communication by transporting data (message) from a sender to a receiver.

How this communication is done, is irrelevant as long as it has the defined core characteristic: it´s one-way.

Yes, I believe not thinking in terms of one-way communication is the main problem of today´s mainstream OO programming languages. That´s where they fundamentally misunderstood Alan Kay. There´s simply too little difference between:

/* Pascal */
Var y : Integer;
y := f(x);

or

// C
int y = f(x);

or

// Java, C#
SomeClass o = new SomeClass();
int y = o.f(x);

Global functions of pre-OO languages became local functions of objects. That´s all? And all of a sudden programming was supposed to become a walk in the park? The maintenance problem solved?

I´m guilty of believing this myself. I thought the combination of data structures and functions into was a godsent. True encapusaltion was possible. And then inheritance and polymorphism… Could it get any better than C++, Delphi, Java, C#? Well, as it turned out, most of the projects wading hip deep in brownfields have been written using these and other OO programming languages. How can that be? Wasn´t object orientation supposed to solve mainteance problem by making encapsulation and reuse so much easier?

Sure, there´s lots of reuse today. And I´m very grateful for “components” of all sorts I simply can download. But has that really been made possible by OO languages? I don´t think so. There were libraries before that era. I even remember switching between C and Modula and use the same libraries. Libraries and then dynamic link libraries brought reuse at our fingertips. VB became so popular because of its VBX-“components” – but the VBX interface definition wasn´t object oriented.

So whatever advantages we´ve gained through reuse is not due to OO in general or OO languages in particular.

Now back to the above code:

I´ve never found any student of programming who really understood why

int y = o.f(x);

was called “sending a message to an object”. It´s a function call. That´s all. And if you remember the early implementations of C++ compilers which compile C++ to C you know why. They transformed an object-local methode like this to a global function call:

int f(SomeClass this, int x) {…}
…
SomeClass o;
…
int y = f(o, x);

So where is the message gone? Instead of sending a message to an object the code is sending data to a function.

A function is a function is a function. Calling it a message does not change that.

So I dare to say: Languages like C++, Java, C# are not object oriented with regard to what Alan Kay originally intended.

They are not, because they heavily rely on functions, i.e. subroutines returning results. Request plus immediate response is the main programming model. Request and response are supposed to happen as close together as possible in terms of time and space. The sender of the request is not just a sender but a client. He´s expecting a result back, right here, right now.

This thinking is so deeply ingrained in our programmer brains, it even led to abberations like RPC and CORBA etc. Think of all the money that was sunk into these attempts to deny the laws of physics.

Backt to messaging:

If we want to take Alan Kay seriously, if we want to truely program in an object oriented way, then we´ve to live by the definition of messaging. That means, there must only be one-way communication. We´ve to do away with functions. Or at least – since functions are useful syntactic sugar for symmetric relationships between message processors – find rules as to how to use this sharp edged sword.

Messaging can be implemented in different ways. It´s not important, if a method name stands for the message type or if that get´s passed to an object explicitly:

o.f(x, …);

or

o.Send(“f”, x, …);

It´s also not important, whether messages are structures or lead to longer method signatures:

SomeMessage m = new SomeMessage{A=a, B=b, C=c};
…
o.f(m, …);

or

o.f(a, b, c, …);

And of course we can continue to use methods as we know them, i.e. pass parameters to them. There´s no need for explicit intermediate data structures or even queues.

Also asynchronous processing is not mandatory for message passing.

I strongly believe the core characteristic of messaging unidirectionality. There is no coupling between request and response. And that means, there are just messages flowing from sender to receiver.

That´s what Steve shows us in his articles. Here´s an example from [1]:

image

Each processing step is implemented as a procedure, e.g.

image

His pipeline infrastructure is doing the message passing work behind the scenes.

I´m not saying this is the best way to do it. But it´s true to the definition of messaging.

And altervative Implementation would be:

var loginPipeline = new PipeLine<LogInMessage>();
CheckUserSuppliedCredentials.Process(msg);
CheckApiKeyIsEnabledForClient.Process(msg);
…
GetUserDetails.Process(msg);

The private instance methods would just need to be set to public static.

Or even simpler:

var loginPipeline = new PipeLine<LogInMessage>();
checkUserSuppliedCredentials(msg);
checkApiKeyIsEnabledForClient(msg);
…
getUserDetails(msg);

Here the processing methods would be defined in the caller´s scope.

Also functions would be ok:

var loginPipeline = new PipeLine<LogInMessage>();
msg = checkUserSuppliedCredentials(msg);
msg = checkApiKeyIsEnabledForClient(msg);
…
getUserDetails(msg);

The pipeline that way would not rely on global data anymore.

And using functions in this kind of controled way is true to the messaging definition: there is no response processing at the caller. Any response just gets passed on to the next processor. This becomes clear when replacing the response with a continuation:

var loginPipeline = new PipeLine<LogInMessage>();
checkUserSuppliedCredentials(msg, _ =>
  checkApiKeyIsEnabledForClient(msg, _ =>
    …
    getUserDetails)
  )
);

Each processing step would be defined like this:

void checkApiKeyIsEnabledForClient(LogInMessage msg, Action<LogInMessage> continueWith) {…}

The source code might not look that straightforward when using lots of continuations, but nevertheless its true messaging passing.

To tame functions there is just one rule: Whoever calls a function does not process the return value itself.

I call that the Integration Operation Segregation Principle (IOSP), but that´s a topic for another article. Suffice it to say, to me that´s a special case of the Single Responsibility Principle (SRP).

But what if a message receiver wants to communicate back something to a sender? Well, then the receiver becomes the sender and the original sender becomes the receiver. Messages of course may flow in both directions between communicating parties. But each message in itself is just one-way.

That´s how communication in the real world works, too. Biological cells don´t know request/response communication. They are autonomous units who even communicate asynchronously. But communication does not need to be async to pass as messaging.

The whole concept of request/response communication is an abstraction. A very useful one as long as the code base is small. But it does not scale. It does not scale, unless reigned in. It fails for remote communication. And it fails if used to build huge hierarchies with logic smeared all over the place.

Technical object orientation – aka features of OO languages – is not evil. Right to the contrary! I don´t want to miss any of C#´s features. But technical object orientation needs principles and rules for use. We should stop programming in an “anything goes” manner. Just because we can do something we shouldn´t do it.

That´s what Steve is shows us. He´s using OO features in a very particular manner (rules) to stay true to a basic principle: messaging. The result is easier maintenance, easier teasting, and code that´s easier to comprehend.

 

P.S. If you liked this, have a look at “Flows – Visualizing the Messaging Programming Model” for a suggestion on how to describe messaging solutions in a source code independent way. And then read on about how to truly decouple processing steps in flows. Ah, by the way: Don´t forget to also check out this one on hierarchical messaging.

 

Endnotes

[1] Steve Bate, Messaging as a programming model – Part 1

[2] Steve Bate, Messaging as a programming model – Part 2

[3] Stefan Ram, Alan Kay, Dr. Alan Kay on the Meaning of “Object Oriented Programming”

posted on Saturday, August 17, 2013 1:59 PM

Feedback

# re: Messaging as a programming model – Let´s get real 8/19/2013 11:08 AM Steve
I think that the conceptual stumbling block that held us back from making OO properly message based, when it became mainstream with C++ and its descendents, is the tacit assumption of synchronous operation -- objects aren't managing their own internal processing for themselves; in the default case they are all managed via the one globally ticking program counter, and at best they simply conceal state. The current rise of syntactic support for asynchrony (in response to multi-core processors) should make it easier to move to a more directly message passing model.

There have been some interesting, if possibly premature, ideas in this line in recent years along the lines of running Ruby or something like that on the Erlang VM with objects mapped to processes -- though they all seem to have been abandoned.


# re: Messaging as a programming model – Let´s get real 8/19/2013 3:15 PM Ralf Westphal
Yes, messaging sounds somewhat synonymous with asynchronous communication. Also Alan Kay´s biology analogy suggests that. But in my view that´s not true. Messaging can be done perfectly in sync scenarios. Like you´re doing it in your examples.

Nevertheless the growing interest in async programming will surely help the messaging programming model.

I myself started out with messaging in the async and distributed world of programming - but then I realized the paradigm could help me with everyday programming tasks, too.

# re: Messaging as a programming model – Let´s get real 8/19/2013 3:41 PM Nick
I'm not seeing the immediate benefit. Or the difference really. I think I need to see an example. A 'bad' before and a better after. And maybe why it's better. Are you saying we should not be calling functions of other objects and getting a return value? But rather the called object should be passed a reference to the caller and send one back?

# re: Messaging as a programming model – Let´s get real 8/29/2013 12:00 PM Akash
I find this topic very interesting, but I don't understand your dislike for request/response. As you say, it is a useful abstraction, and as long as the req/resp is implemented in a non-blocking async manner, it should scale just fine.
Can you explain further? Thanks!


# re: Messaging as a programming model – Let´s get real 8/29/2013 3:19 PM Ralf Westphal
Request/response processing tends to couple two aspects: whatever happens before a request and whatever happens after receiving the response.

Technically that´s fine. Like shooting yourself in the foot.

But using the tool without principles/rules hurts understandability and evolvability very quickly. Because since request/response is so easy you can do it any number of times in a subroutine. That leads to a mixture of logic and delegation. And that leads to huge subroutines.

Being able to do request/response in an async manner (like with the C# async/await keywords) does not bring relief. I´d even say it makes matters worse.

# re: Messaging as a programming model – Let´s get real 8/30/2013 10:19 AM Nathan Evans
Interesting topic.

It feels like req/resp is frowned upon more for its greater ease of which it can be misused compared to a pure "pipeline pattern" aka GOF's chain of responsibility.

Seems there's a number of obvious ways req/resp can be, including:

* bundling large amounts of "request setup" code before the actual request is sent, in the same method

* bundling large amounts of "response handling" code after the response is received, in the same method.

* performing multiple req/resp operations in a single method

I was working on a pet project last weekend. I had my web project's MVC controllers doing a little request setup (typically nothing more than unwrapping a model object and passing those params into the ctor of a request object), performing a fairly nice async/await to do the req/resp operation, and then handling the C#-emulated-DU result object that came back. None of my controller methods performed multiple req/resp operations.

On the server side, it would receive the requests of a Azure Service Bus, process them through a pipe not too dissimilliar to what is presented in this article, and ultimately send back a response.

In my experience, developer time is the most important thing here. You have to "triage" the codebase, especially a greenfield one where iterating quickly is important. I always prefer to keep my "server-side" code in better health than the "client-side" code. Yes, I consider the MVC web app in this to be client-side code. In the grand scheme of things, it is. It doesn't have the same reliability and performance constraints usually. That said, I'd never allow the client-side code to get so woefully unmaintainable in relative comparison to the server-side. There is a balance to be found for sure. Most importantly, the req/resp client-side code can be refactored in the same ways as anything else. It's not set in stone, especially on a greenfield project.

TL;DR: I don't think there is anything inherently wrong with request/response as a pattern. But like all patterns it can be abused, whether willingly or unwittingly. Req/resp is a good pattern to use on client-sides, where project iteration speed is important.

@nbevans

# re: Messaging as a programming model – Let´s get real 8/30/2013 10:49 AM Ralf Westphal
As I said: req/resp is a tool. It´s once way to translate certain parts of a design.

But it needs to be reigned in. That´s not what happens usually. People use req/resp indiscriminately - which easily leads to subroutines of thousands of LOC and code with logic scattered all over the place.

# re: Messaging as a programming model – Let´s get real 8/30/2013 11:04 AM Nathan Evans
All patterns can be abused. Even the pipeline/chain of responsibility pattern. I.e. a lazy dev would simply add more responsibilities to an existing one than actually, you know, create a new responsibility and add it to the correct stage in the pipeline.

I'm not really follow your reasoning. In your article's example of a server-side pipeline performing various stages of authenticating a logon request... that's all fine. But what happens when the pipeline outcome is "authentication successful". It needs to return that indication to the client application (i.e. a MVC web app). How would you do that? You'd of course use a Correlation Id, of some description (whether leaning on your service/message bus, or rolling it yourself, doesn't matter), and then you're back in the realms of request-response.

# re: Messaging as a programming model – Let´s get real 8/30/2013 11:10 AM Akash
So would request/request be acceptable if its use was limited to n (you choose a value for n >=1) calls within a subroutine (let's pretend we had a smart compiler that enforced this)? Or do you see it as a fundamentally flawed approach?

If it is fundamentally flawed, then I think an example of how you would avoid it would be very useful.

# re: Messaging as a programming model – Let´s get real 8/30/2013 12:13 PM Ralf Westphal
@Nathan: First of all, the pipeline example is referring to some code written by Steve Bate. I´m using it just to illustrate different variants of implementing flows. This is supposed to keep the discussion of messaging somewhat more cohesive.

Apart from that you seem to have a premise in the back of your head which I don´t have, that of async communication. Because only then mentioning correlation make some sense, I´d say. But there is no async communication implied anywhere in my current writing on messaging. If you look closely at my code, the non-pipeline code, then you´ll see there is just sync communication happening. One way or the other.

Finally: Just because in the end some functional unit receives input after it issued some output does not mean req/resp is employed. It might be a subtle difference, but to me it´s a very important one: when doing a request the requestor is waiting for/expecting a response. But that´s not the case if some UI fires an event. It´s just, well, some output - without any expectation of when/if something flows back.

# re: Messaging as a programming model – Let´s get real 8/30/2013 12:16 PM Ralf Westphal
@Akash: I don´t want to limit the use of req/resp to any n. You can call 42 functions from a subroutine, if you like (and if you still understand what those calls are supposed to accomplish as a whole).

What´s important to me is what else is happening in such a subroutine. For that have a look at this article: http://geekswithblogs.net/theArchitectsNapkin/archive/2013/08/19/nested-messaging---flows-on-different-levels-of-abstraction.aspx

Call as many functions as you like - as long as you don´t put any logic in the calling subrotine.

# re: Messaging as a programming model – Let´s get real 8/30/2013 12:51 PM Akash
Ok, I think I'm beginning to understand. You are against mixing sequencing with operations within the same subroutine. I can buy into that.

On your comment to @nathan, I think it *is* important to reify the req/resp pattern when the communication is truely 2-way, rather than relying on events. For example, when a component sends a login request, it *is* expecting a response, therefore there should be explicit support for that within the programming model; also, the responder should be forced to guarantee a response (i.e. it should implement an appropriate interface).


# re: Messaging as a programming model – Let´s get real 8/30/2013 2:06 PM Ralf Westphal
@Akash: Well said, no mixing of sequencing with operations in the same subroutine. I call that the "Integration Operation Segregation Principle".

As for req/resp: The question is: Is 2-way comm. really needed?

Look at this:

void client() {
// before
var y = service(x);
// after
}

There is 2-way comm. required here, it seems. The client sends data to the service, and the service returns data to the client.

Now look at this:

var x = client_before();
var y = service(x);
client_after(y);

The 2-way comm. has evaporated! There is no request anymore. There is no response processing anymore.

client_before() just produces some data. client_after() just processes some data.

And both are easier to test than the original client().

So I´d say, req/resp thinking is not a requirement but a choice. And I choose to live without it as much as I can. Live has become much easier since than.

# re: Messaging as a programming model – Let´s get real 8/30/2013 5:31 PM Akash
@ralph, I think we disagree on the meaning of 2-way comm!

var y = service(x) is present in both of your examples, and I see that as 2-way comms i.e. the component containing that code is sending a message to the service and *requires a response* in order to be able to continue.

The rest of your second example is to do with splitting up the logic that happens before and after the 2-way comm, and could be good/bad depending on the statefulness of the client.

# re: Messaging as a programming model – Let´s get real 8/30/2013 6:55 PM Ralf Westphal
I don´t think we have a misunderstanding as to what a 2-way comm. is. If C calls S and expects a result back, then there is a 2-way comm. between C and S.

Now, I´ve shown that C - consisting of "before", service call, and "after" - can be dissolved. There is no C anymore which does a request and/or awaits a response. There are now two distinct functional units (client_before() and client_after()) which in sum (!) do the same as client() did. But neither calls service(). For all practical matters there is no 2-way comm. anymore between client functionality and service().

Whether client() was stateful or not is of no concern for this. State can be retained between client_before() and client_after(). That´s what objects are for.

The interesting part, though, hasn´t been commented upon by you ;-) That´s the encompassing functional unit which calls all three subroutines. It´s anonymous in my code example, but it´s there nonetheless.

If you like, call it clientV2() ;-) Because it accomplishes what client() formerly did. But it´s focused now on just 1 responsibility: integration. And that makes a whole lot of difference in my view.

client_before(), client_after(), and service() are composable function units. No functionality is lost. But composability is gained. And understandability. Because whatever was "before" and "after" now had to be named (think of the SLA principle).

It´s easy to test just client_before() and client_after(). The integration does not need a test of its own; it´s trivial.

Structuring you code according to the IOSP makes it easier to read and easier to test and easier to reuse. Subroutines tend to become much smaller, because you can´t just pile up logic and subroutine calls in them.

Just follow a simple rule: If you put logic in a subroutine, don´t call other subroutines of your own. (Calling APIs is ok.)

# re: Messaging as a programming model – Let´s get real 9/12/2013 9:10 AM Witold Szczerba
Hi,

I am thinking about adapting the PipeLine approach to see if I can gain code radability and maintanence.

There is one thing I don't get. Each filter is parametrized with type T which has to be common to the entire process.

How can we create filter for common tasks, like for example to fetch user details, if each time the message type would be different?

Are you creating common interfaces (like UserDataContainer) and make the message type implement them, so the same filter can be reused anytime when T implements UserDataContainer?

Or the answer is not to reuse the filters, but use each time new one, to wrap the typical UserService or UserDao, containing methods like getUser() : UserDetails?

# re: Messaging as a programming model – Let´s get real 9/12/2013 12:13 PM Ralf Westphal
To be honest, I´m not that a big fan of using the same message type for all filters. That´s an idea Steve Bate came up with, and I used it to make it easier for readers of his blog posts and mine to follow them.

But on a more general note I´d say don´t try to thing in terms of reusability too early. That´s an orthogonal concern, though.

# re: Messaging as a programming model – Let´s get real 9/12/2013 3:23 PM Witold Szczerba
Hi again,

> To be honest, I´m not that a big fan of using the same message type for all filters.

I was thinking about this as well, but how to implement filters with different message type? Where would we adapt message type before passing into filter?


>But on a more general note I´d say don´t try to think in terms of reusability too early. That´s an orthogonal concern, though.

The question is not just about reusability, but about the core concept of the pipeline and filters. How to organize code? If two filters are supposed to do similar job, it's not an option to copy/paste code between them.
You are right I should have asked Steve, because he was the one who blogged about pipe and filters and used them, my bad.

I would really like to see some code structure, I mean the big picture, not code snippets, using extensive messaging model using pipe and filters or something else. Do you have any references? By the way: did you read Growing Object Oriented Software, Guided by Tests, by Steve Freeman and Nat Pryce? They highlight the OOP, as most of us think, has little to do with Alan Kay's original idea and promote thinking about messaging as a way for objects to comunicate. Do you have any opinion on that book?

Thanks,
Witold Szczerba

# re: Messaging as a programming model – Let´s get real 9/13/2013 9:50 AM Ralf Westphal
Working with different message types to me is very natural. Just define each filter with its own signature - then wire them up appropriately :-) That won´t work with Steve´s pipeline, though. But still it would be messaging.

These articles are not about the big picture - but there is one ;-) A concrete one for a specific app, and one on the meta level. Maybe I find some time to write about it. (I could point you to some articles on it in German, if you like ;-)

Messaging is mentioned in the GOOS book - but other than that I don´t really see a real implementation of it. There is no definition of messaging. So they stick to SOLID as far as I can see. They simply restate broad brush statements about messaging we can read elsewhere.

Unfortunately I don´t like the book as much as many others do. Great intention - but then there are gaps in the design process. I simply don´t believe we should rely so much on TDD. TDD/tests are important - but not a "solve all" for design questions.

Post A Comment
Title:
Name:
Email:
Comment:
Verification: