The last couple of weeks have seen a significant increase in terms of announcements and information from Microsoft ahead of the Professional Developer’s Conference next week. It is a key time for Microsoft’s Connected Systems Division (CSD) as they go public with their plans for .NET 4, Oslo and ‘Dublin’. Microsoft has also announced the forthcoming release of BizTalk Server 2009 in the first half of next year. The public ‘messaging’ for all this new technology has been a hot topic of debate within CSD and its technical advisor community during the last year, and it is vital that Microsoft paints a coherent, plausible and commercially attractive picture of its plans and directions over the next fortnight. Early indications are looking good. A lot of feedback has been provided, and CSD has clearly been listening. Hopefully the benefit of all that effort will pay off at the PDC.
More importantly, perhaps, the technology is looking good. The first public view of Oslo (modelling tools) and .NET 4 will be provided in the form of a CTP (Community Technology Preview) disk to PDC attendees. No doubt a more public CTP will follow in due course. It’s a little too early to discuss the technology in detail, but you should expect to see a flurry of blog articles appearing post-PDC.
What I want to do, here, is to pick up on the announcement of ‘Dublin’ made a couple of weeks ago (see http://www.microsoft.com/net/dublin.aspx), and talk about the relationship between Microsoft’s forthcoming application server and BizTalk Server. Again, all technical detail must be avoided for the time being until MS is ready to go public. However, Microsoft’s announcement led immediately to an interesting, if somewhat predictable, debate within the company I work for, and I think it is worth going over some of that territory here. Needless to say, anything I write here is completely without mandate or scrutiny from Microsoft, and in no way represents an ‘official’ point of view. I don’t work for Microsoft!
Introducing .NET 4
To recap, Microsoft announced ‘Dublin’ as part of a more general announcement regarding .NET 4.0. Dublin is a .NET 4.0 technology which will be released in due course. Although the announcement falls short of specifying any licensing arrangements, it is clear that Microsoft considers Dublin to be a set of extensions to the operating system which will be delivered as part of Windows in future releases. It seems, therefore, that it will be tied to Windows licensing. I have no idea if it will be available for use with all versions of Windows. However, the way it is being pitched suggests that it may well be – in much the same way that IIS and COM+ are available across both server and desktop editions of the OS. This is, of course, speculation on my part.
The list of features for WCF and WF in .NET 4.0 can be obtained from a document available for download from the above link. It’s worth taking a look, if you have not already done so. There are several points of interest, including:
a) A major push to support RESTful interaction. This is surely a welcome evolution of WCF which previously has majored chiefly on SOAP-based interaction. I have never felt any need to take sides in the debate on REST vs. SOAP. I do, however, follow that debate with some interest. As a BizTalk developer, of course, I have expended more effort in recent years on building SOAP interactions. I was more RESTful in a previous life. I remain of the opinion that SOAP has a lot to offer the world of enterprise computing, and is an appropriate choice in many scenarios. However, the lack of good support and tooling for RESTful interaction in BizTalk-land means that we certainly overuse SOAP. More importantly, I am convinced that REST will play a central and vital role in extending the utility of service orientation across ever wider territory.
b) Much closer (‘seamless’) integration between WCF and WF, centred on XAML-only development. WCF is Microsoft’s framework for loosely-coupled service interaction. WF is a complementary framework of tightly-coupled activities, state transitions and data flows. As our understanding of service orientation grows, it becomes ever clearer that we need to address the certain aspects of service implementation, such as data flow, in as rigorous a fashion as the definition of loosely-coupled service interfaces. BizTalk developers understand first-hand the benefits of declarative, model- driven development of internal service implementations thanks to BizTalk’s orchestration, pipeline and mapping tools. It is high time that this approach, pioneered by Microsoft years ago within their enterprise products, becomes generally, and generically, available to a much wider community of Windows/.NET platform developers.
c) A much re-worked version of WF. At this time, it would be wrong to go beyond the feature list Microsoft provides in the document I mentioned. There is a lot of technical detail to be announced at the PDC. It is sufficient, for the time being, to say that CSD has expended a lot of thought and time over the new version of WF. There will doubtless be some pain involved for existing WF developers (change always introduces a degree of pain), but the pain is necessary and beneficial, and raises WF from the level of ‘interesting’ and ‘useful’ to the level of ‘vital’ and ‘central’. I always tended to perceive WF as somehow ‘secondary’ to WCF. This is no longer the case.
Dublin and BizTalk Server
At a Microsoft conference, earlier this year, I sat alongside a fellow BizTalk consultant during an interesting open forum on Microsoft’s Patterns and Practices offerings. We were probably the only BizTalk developers amongst the 40 or so people in the room. Very quickly, we both realised that our BizTalk-centric viewpoint was quite distinct to everyone else’s – and so we had the good sense to keep reasonably quiet. One of us (I forget who) raised a question about the ESB Guidance Toolkit, which is part of P+P, but no-one else seemed to be aware of it. We ended up whispering conspiratorially like two schoolchildren on the back row!
The interesting thing for us was that the pain-points developers highlighted in implementing web-based applications and services, desktop applications, etc., centred on the lack of frameworks, guidance and tooling needed to address loosely coupled interaction patterns and distinguish between these and tightly coupled data flows. Our whispering mainly consisted of suggesting ‘BizTalk’ under our breath as a universal panacea for the problems being raised. Of course, we didn’t mean this literally. We were simply struck by the fact that the issues that developers were discussing in that forum have broadly been addressed for years by BizTalk Server at the level of enterprise integration and business process automation. What most of the developers in that room needed was platform-level guidance, tooling and framework code that supported synchronous and asynchronous interaction patterns and practices in a form suited to a wide spectrum of scenarios and application types.
I’ve often thought back to that forum session. It is not often that you get to hear such clear, consensual, feedback on where the wider world of development is at. What was, perhaps, not clear to most people in the room, is that their world is set to become very much better.
Dublin, building on the .NET 4.0 platform, is a major piece in an emerging set of new and updated technologies that Microsoft will ship over the next few years. Alongside ‘Oslo’ (the modelling and repository tools that Microsoft will launch in a few months), and what, hitherto, we called ‘BizTalk Services’ (support for extending the service bus into the ‘cloud’), Dublin will provide a core set of development and hosting facilities that will greatly extend the utility of WCF and WF and will reduce the cost and complexity of building robust and scalable applications on the .NET platform. However, if you read Microsoft’s published feature list for Dublin, it sounds a lot like BizTalk Server. Dublin will provide hosting facilities, message-based correlation, message forwarding services, content-based routing, compensation of long-running transactions, scalability features, state persistence and rehydration, management and monitoring functions and an event tracking store. You could be reading promotional material for BizTalk Server!
The questions that arise naturally from Microsoft’s published feature list are urgent and pointed. Is BizTalk Server, as we know it, to be replaced by Dublin and relegated to the ‘legacy’ back shelf? Where would that leave existing customers who have paid serious money to purchase licences, and even more serious money to develop BizTalk solutions? If BizTalk Server still has a role, how is this differentiated from Dublin, and how do we articulate this meaningfully to potential customers? Why would anyone want to buy BizTalk Server licences in future?
It’s a difficult situation for Microsoft to handle. However, after months of internal debate, and much to my relief, CSD has hit on a clean, simple and useful shorthand to communicate the difference between Dublin and BizTalk Server. They describe Dublin as an ‘application’ server, and BizTalk Server as an ‘integration’ server. Of course, this needs plenty of qualification, but it offers simple mental hooks for people who need to understand the difference. More than that, CSD is keen for their customers to know that they remain highly committed to BizTalk Server. This well-established product has a whole road map of its own with a new version due next year and plans for a major release sometime ‘post-Oslo’. Dublin is not a competitor product. Existing BizTalk customers won’t be abandoned, or left with an expensive white elephant. Dublin won’t undercut the core value proposition of BizTalk Server.
We have seen the term ‘application server’ used before within the Microsoft world. In the 1990s, development on the Microsoft platform largely revolved around the Component Object Model (COM). I feel a strange urge to spend time explaining what COM is for those who don’t know – a mark, perhaps, of how far we have travelled along the .NET road in recent years. I will resist doing that, but will note that the core technology that became the .NET CLR originally existed as the implementation of a runtime environment for a new version of COM. Microsoft even had a name for this new technology. They were going to call it COM+.
History, of course, took a different turn. The new generation of COM technology was largely abandoned in favour of a major re-work which resulted in what we now know as .NET. The term ‘COM+’ was recycled, and emerged in 1998 as the new name for a much extended version of an earlier technology called MTS. COM+, then, is the name we give to Microsoft’s ‘application server’ for COM and which remains an integral part of the Windows platform to this day.
When you think of Dublin, think of COM+. Don’t take this too literally, though. COM+ was designed as an application server for the RPC-based world of the 1990s. Distributed application components communicated through object-based interfaces on local proxies generated over type libraries using low-level communication based on the familiar semantics of procedure calls. COM+ provided many services designed to support the implementation of enterprise-level COM-based applications. These included distributed transaction control services, load balancing, discovery and activation services, management services, asynchronous invocation, resource management, enhanced security control, a loosely coupled event broker and many other features.
Dublin is the Windows-based application server for WCF and WF. Instead of object-based RPC, Dublin lives in the service-orientated world supported by WCF and WCF and defined chiefly by SOAP and RESTful interactions. Re-read the list of services it provides from this perspective and you should begin to understand the scope, purpose and nature of Dublin, and its role as part of the operating system. It is also the explanation for why Microsoft describes Dublin as ‘complementary’ to BizTalk Server. To be exact, Microsoft describes their ‘workloads’ as complementary, indicating that they expect the two technologies to be used side-by-side to handle different types of interchange and message traffic.
Defining the BizTalk Perspective
I think it was Clemens Vasters who once described the BizTalk mapping editor as the ‘killer’ feature of BizTalk Server 2000. Clemens is now a Microsoft employee, and the person who, if he but knew it, first got me hooked on BizTalk Server. His point, if I recall correctly, was that, although BizTalk Server 2000 was a rich and multi-faceted piece of software (though not a patch on later versions, of course), the bit with the real ‘wow’ factor was the mapping editor. That’s what got people’s attention back in 2000. It is what ‘sold’ the product.
The mapper, of course, was never the truly defining aspect of BizTalk Server. Perception and reality often diverge. I believe that the advent of Dublin will mean that Microsoft, and BizTalk consultants like myself, will have to think deep and hard about how we communicate the value proposition of BizTalk Server to potential customers in the future. People will naturally ask why they need to purchase BizTalk Server licences when Dublin apparently offers similar functionality ‘for free’. What is it that makes BizTalk Server worth buying?
Here are some initial thoughts on differentiation between the two technologies, and they are not actually that profound:
Differentiator 1: It’s the Message Box, stupid.
For ages, the BizTalk community has been pleading with Microsoft to provide some way of doing integration and message routing in BizTalk that bypasses the message box. CSD long since got the message (no pun intended), and Dublin, I suspect, points the way forward. Of course, what we really want is to be able to combine low latency approaches seamlessly with the rest of BizTalk Server. It will be interesting to see what Microsoft does in some future version of BizTalk. In the meantime, we need to remember that, despite its obvious latency issues, the BizTalk message box is a Good Thing.
Look again, very carefully, at the published feature list for Dublin. There is talk there about persisted state, curiously related to scalability rather than resilience. There is no talk, however, about persisted message queues and the ability to resume suspended message interchanges. I will speculate here that a major differentiator between BizTalk Server and Dublin will be BizTalk’s persisted, transacted queues with their ability to differentiate between, and efficiently handle, service instances, messages, message context and message content. This is surely the core feature of BizTalk Server, and the foundation on which so much of its value proposition rests.
Take the project I am currently working on. A global manufacturing company is integrating legacy ERP systems used in different parts of the world with a new ERP system that will progressively take over from those legacy systems. This will not happen overnight, and so the company has an entire program, spanning a significant time period, in which they are evolving an integrated solution that supports bi-directional synchronisation of master records between the old and new worlds. Eventually, everything will move to the new world, but for the time being, a range of different record types, including high volume GL data, must be exchanged efficiently, reliably and robustly between different applications, platforms and locations.
Although the mechanics of the BizTalk solution are relatively straightforward, this is no trivial task. I’ve only been working on the project for a few weeks, but in that time, I have seen several different issues arise; malformed messages, unexpected changes within ERP systems, problems regarding communication with external queues, unstable services that don’t manage session state cleanly, etc, etc. These problems generally lie beyond BizTalk Server’s control, but BizTalk Server lives at the strategic heart of this distributed system. It ‘sees’ the whole picture. It handles the exceptions that arise in external systems, but which affect the interchange of messages between systems. It is, therefore, central to investigating, troubleshooting, managing and monitoring the integration of the various ERP systems. It plays a vital role in recovery from failure, even though it is not necessarily the cause of that failure. This is what it means to be an ‘integration’ server. Dublin will not play an equivalent role in these kinds of scenarios.
So, differentiator one is the message box, and the message box is Good.
Differentiator 2: Vote XLANG/s for president
This is a more difficult argument to make, and one that may not convince every reader, but I do have the courage of my convictions. First, to dispel a dangerous myth. We should not expect not see the existing XLANG/s orchestration technology replaced by a new WF-based orchestration engine in future versions of BizTalk Server. This would be a terrible idea at many levels, even though it sounds both obvious and beguiling. We may well see new features that allow us to use WF more effectively alongside XLANG/s (pure speculation on my behalf, incidentally), but XLANG/s is set to remain a central part of the BizTalk Server story. Why? Well, first, there is a commercial consideration. How would you feel if, having invested large amounts of money in building sophisticated solutions using XLANG/s, Microsoft ripped and replaced this technology with something that exhibits a similar feature list at the top level, but which is a fundamentally different technology. Forget pie-in-the-sky ideas of building a WF version of orchestration that painlessly runs existing XLANG/s scripts. XLANG/s is an enterprise-level technology built on a sophisticated step management engine which is deeply integrated with the BizTalk Server message box and message agent. WF, by contrast, is a developer’s framework built on the concept of data flows handled by ‘activity’ object graphs. Although there is a great deal of commonality, they actually operate at somewhat different levels of abstraction. I do not believe it is technically feasible to achieve a painless migration path from XLANG/s to some future WF ‘equivalent’, and I certainly would be deeply unhappy if Microsoft compromised the extensive investment in XLANG/s made by so many customers over so many years. CSD, incidentally, understands that argument.
Of course, one of the most attractive aspects of WF is its tremendous extensibility. This is a feature that XLANG/s does not share. Nor, in my opinion, should it. Contrary to popular opinion, the core value of the XLANG/s technology does not reside in the orchestration designer, or the somewhat limited library of ‘shapes’ that it supports (roughly analogous to WF activities). It resides in the painfully precise and comprehensive semantics and rules built into the XLANG/s ‘compiler’ – rules which are founded on a solid basis of computer science (rooted historically in pi-calculus), and which are enforced rigorously throughout the technology, often to the general dismay of BizTalk newbies. They cannot be broken by poor programming technique or shallow understanding of the complexities of service orchestration, and are deeply integrated with BizTalk’s notions of asynchronous message routing via subscription. I would certainly like to see XLANG/s improved. The designer is long due an overhaul. There must surely be scope for a richer set of shapes. I do not, however, want WF-style extensibility because I do not believe it could be provided without disastrous compromise.
The problem with BizTalk orchestration is that it is so useful that it has been too widely used for purposes other than orchestration in the absence of any alternative. WF provides that alternative, and is generally the better choice for non-orchestration purposes. When it comes to robust, dependable, enterprise level orchestration of interchanges between different services, systems and applications, XLANG/s is the technology to back. We don’t need WF to replace XLANG/s. What we need is better synergy and seamless, safe, interaction between the two technologies. That, I believe, is the future path that BizTalk Server should take.
Differentiator 3: Stronger regulation needed
This has to be BizTalk’s hosting and administration story. Now, with regard to the future, much is still under wraps. However, consider the following. Today, BizTalk Server offers single-point administration of two distinct host environments – IIS worker processes (for so-called ‘isolated’ BizTalk hosts) and BTSNTSvc.exe processes. BizTalk Server builds on this to provide a sophisticated ‘cluster’ approach based, not on Microsoft’s Cluster Service, but on forms of load balancing across multiple stateless servers, driven off a central set of persisted queues. This load balancing approach contains complex plumbing to manage different types of throttling and load management, dynamic introduction and removal of servers from BizTalk groups, and many other features that allow organisations to manage a range of complex and often non-deterministic operational requirements. This can only get better in future, but do not expect to see this level of sophistication provided ‘free of charge’ with Dublin! Dublin, in any case, will major on offering other, synergistic approaches that will, I believe, prove very welcome to BizTalk developers as well as a much wider community.
Differentiator 4: Investing in the market
Differentiator four picks up on a very obvious trend in the current versions of BizTalk 2006/2009. Microsoft has added considerable value to BizTalk Server over recent years by including functionality aimed at specific markets and industries. Examples include the greatly extended support for EDIFACT and X12 in BizTalk Server 2006 R2, the inclusion of RFID Server, HIPPA support (not generally relevant on this side of the pond) and several new features planned for next year’s release. Microsoft includes these extensions in order to drive additional value off the core product, and to extend the reach of BizTalk Server into verticals. This kind of approach is not likely (an understatement) to be a feature of the Dublin story.
Differentiator 5: Putting Lipstick on Pigs (oh come on, give me a break...)
This differentiator is the likely tooling story for the two technologies. Being a typical developer, I am never, never satisfied with the tooling we are given. I have a long wish list for improvements to BizTalk tooling (how about resizable code editor windows in the orchestration designer, for starters). However, I suspect that tooling will be a major differentiator between Dublin and BizTalk Server in the future. It has never been Microsoft’s way, in the past, to provide rich tooling over base platform services and frameworks. By contrast, and going back to Clemens Vaster’s comments, tooling has always been a major part of the BizTalk Server story. I suspect I am on solid ground if I speculate that this will continue to be the case in the future. Of course, the arrival of Oslo modelling tools will mark a fundamental shift in gear with regard to tooling for the Microsoft platform as a whole, and will doubtless impact both technologies greatly. However, I suspect that BizTalk Server will continue to offer superior development and administration tooling, and will expolit Oslo deeply (as, we are told, will Dublin). Again, this is pure speculation on my part. I do not have any insider knowledge of detailed plans.
I will not be at the PDC next week, unfortunately, but several of my colleagues from SolidSoft will be, so if you are there, look out for them.