Charles Young

After publishing the article on CEP on the Microsoft platform a couple of days ago ( see http://geekswithblogs.net/cyoung/archive/2009/05/18/cep_explained_for_biztalk_users.aspx), I was reminded of an excellent presentation given by my good friends John Plummer and Jeff Johnson at the Microsoft Architect Insight conference in 2008.   John and Jeff both work for Microsoft in the UK.   They used NEsper (the .NET port of the open source Esper event processing engine) to demonstrate how CEP might be used in conjunction with BizTalk technologies.   I dug out the slides and reminded myself of their presentation.   All very good stuff.  the slides are at:

http://download.microsoft.com/documents/uk/msdn/events/SOL/SOL08.pptx

Wow...three posts in one day - a world record for me.

According to Brian Loesgen, ESB Toolkit 2.0 for Biztalk 2009 will be released mid-June.

Last Monday, Microsoft used the TechEd keynote to announce their plans to include a Complex-Event Processing (CEP) engine in SQL Server 2008 R2.   This article attempts to explain CEP from the perspective of a long-time user of BizTalk Server and to address the question as to why BizTalk architects and developers might have a reason to take an interest in the forthcoming CEP engine.

Read on at http://geekswithblogs.net/cyoung/archive/2009/05/18/cep_explained_for_biztalk_users.aspx...

Following on from the news about Microsoft's CEP engine, Richard Seroter attended yesterday's TechEd session on the new engine and has blogged very detailed notes.   See http://seroter.wordpress.com/2009/05/12/teched-2009-day-2-session-notes-cep-first-look/.    I'm delighted to see that Microsoft is disclosing much the same level of information publically that they made available internally a couple of months ago.  Now all we need is CTP 1.

Richard notes that the session was sparsely attended.   CEP has some way to go before it is widely acknowledged or understood in the mainstream, and the potential impact of engines like this will be poorly understood at present.   I intend to put together an article or two in which I'll attempt to explain why I think we should all take note.

I responded to a comment from Tim Bass last night.   Tim is a vocal critic of the CEP marketplace, and one of his common themes is the belief that most/all CEP vendors currently attach the term 'CEP' incorrectly to what are 'mere' event stream processing engines.   I responded by pointing out the CEDR technology, on which I believe Microsoft's engine in based, directly tackles one of the chief characteristics of the event 'cloud' (a core concept in CEP) which is that events may be detected out-of-order in terms of time and causality, and that CEP engines must be able to handle this efficiently.   I was interested to see that Richard recorded an emphasis on this capability of Microsoft’s new engine.

Also, one other point.  I had a Twitter message yesterday from another well-known person on the CEP circuit who said that he hadn't seen any sign of anyone using the new Microsoft engine yet.  For the avoidance of confusion, Microsoft only announced the engine yesterday.   There is a very early private preview of the code doing the rounds internally within Microsoft.   It looks like the first public CTP will be available later this year and that launch is expected next year.   So no one is yet using the engine, apart from Microsoft themselves (apparently, if I understood comments made during the keynote correctly, they are currently handling 500 million events a day through the engine in association with their web site).

NewsFlash...Microsoft today announced their long-awaited entry into the Complex Event Processing market.   The press release is at http://www.microsoft.com/presspass/press/2009/May09/05-11TechEd09PR.mspx.   The news was also announced by Bill Veghte as part of his keynote at TechEd.   The Microsoft CEP offering will be delivered as part of SQL Server 2008 R2 

Unfortunately, this is the only information I can find so far in the public domain, and, being a cautious kind of guy, and NDAd up the hilt, I will hold back from saying more until I can guage just how much detail Microsoft is releasing at the current time.   It looks like they have a session at TechEd tomorrow on their forthcoming offering.   There was talk a few weeks back of an early public CTP, but there is now some suggestion that this may not be available until later in the year.   Watch this space for more info.

In the meantime, here are a couple of Microsoft Research links that you might find interesting.   I couldn't possibly say how relevant they might be ;-)

http://research.microsoft.com/apps/pubs/default.aspx?id=70517

Over on the ILOG blog, Chris Berg called out Andrew Siemer’s post at http://geekswithblogs.net/AndrewSiemer/archive/2009/03/30/ilog-rules-for-.net-3.0-ndash-quick-overview.aspx.   I thought I’d post some observations by way of a response.

See http://geekswithblogs.net/cyoung/archive/2009/04/29/131593.aspx...

I've been engaged in some debate in the last few days with Jean-Jacques Dubray and Doug Purdy with regard to Oslo.   See:

http://www.ebpml.org/blog/182.htm

http://www.douglaspurdy.com/2009/04/09/where-is-oslo-going-part-ii/

Jean-Jacques has questioned aspects of Microsoft's strategy.   Following some issues with posting anything but the shortest of comments on the ebPML web site, I'll follow Doug's lead and post a response here.

I strongly agree with Jean-Jacques' statement that 'M3 exists'.   It seems a little strange to feel the need to state this emphatically, but the existence of M3 has sometimes been challenged.   Today, 'Oslo' does little to call out or underline the existence of M3.  'Oslo' really isn't expressed in terms of 'classic' metamodel architecture.  I would expect this to be unnerving to some people in the MOF community, and my interest, in a previous post, has been to attempt to show that, despite appearances, there is actually an underlying and solid foundation of common ground between OMG specifications and Oslo.  There is also a lot of history to this.

It seems to me that multiple metamodel layers exist regardless of technology or specification.   However, the degree to which we need to express them within any given context, and the way we choose to do so, is absolutely a matter of pragmatics.   I'm schooled in the old linguistic philosophy that languages (including modelling languages) are built on the 'trinity' of syntax, semantics and pragmatics.   It is these three areas that Oslo so directly calls out and emphasises.   To date, it has, if anything, overly-separated these distinct concerns and I suggest that a lot of the feedback Microsoft has received from the CTP is, in reality, asking them to look for more effective and practical interaction between these different aspects of their emerging technology.   These requests are absolutely driven by pragmatic considerations.  So, while I appreciate Jean-Jacques' comments about pragmatics, I think that this is potentially the area of greatest strength in terms of what Microsoft is doing.   If they get it right (and there is no guarantee of that), they will succeed in providing a set of tools that add real and significant value to their platform and support far more effective use of models in both architecture and development.

In terms of the technology, Oslo offers one language - 'M', founded on graph theory and expressed through MSchema and MGrammar.   M is designed to be used at any level within a metamodel architecture.   I've always found it difficult to imagine any practical need for M4 or above, but I guess you could extend this indefinitely!   What Microsoft doesn't do in the current CTP is to provide any pre-canned M3 meta-metamodels (actually, that is not entirely true - there is the beginning of some M3 support in the repository).   Given that they plan to deliver fulsome support for UML, and have committed themselves publically to XMI, and given their OMG membership, it is possible that we might even see Microsoft exploit MOF.   If they don't do this, themselves, it will only be a matter of time before others do within the Oslo ecosystem.   What I don't expect to see is any attempt to promote MOF as a one-size-fits-all M3 specification in the world of Oslo.   That should not be interpreted as some conspiratorial (and completely pointless) attempt to undermine well-accepted standards.   It is simply the natural consequence of remaining agnostic with regard to the number of metamodel layers needed, pragmatically, within any given context.

Jean-Jacques engages is an interesting argument about 'defining M3 properly' for implementational purposes.   There is a lot in what he is saying.   MOF was originally bound to CORBA IDL, and Microsoft's RTIM (an ancient specification that was roughly equivalent to MOF) was similarly bound to COM IDL.  The first thing that happened when Microsoft put their specifications into the hands of a standards organisation was that RTIM was rejected as being too proprietary.   Similarly, but more positively, the OMG has rigorously decoupled MOF 2 from CORBA.  Jean-Jacques is, I think, suggesting that MOF still fails to adequately handle the kind of impedance mismatches that can occur when creating bindings to very different run-time representations.   I will leave it to others to argue the finer points of MOF, but in terms of Oslo, the stated intention is to reduce the barrier between models and runtimes to the point of near-invisibility.   The Oslo goal is to promote the direct consumption of models (including metamodels and meta-metamodels) within a wide variety of runtimes.   No doubt, as Oslo evolves, we will see the advent of tooling and technologies that help developers to build the widest variety of model-driven run-times.   I have a suspicion that XAML will end up being central to much of this effort.

To summarise, M3 is always there in some sense, but doesn't always need to be represented explicitly within a given context.   Microsoft's historic experience of building multi-level metamodel architecture and repository technology strongly indicates that if you try to force explicit representation of M3 on your development and ISV communities, they will often ignore what you offer.   Oslo does not require MOF compliance or any alternative, but is perfectly capable of supporting this as required.   The ability to bind model representation at any appropriate layer of a metamodel architecture to runtime environments in a thoroughly pragmatic fashion is likely to be a core decider in terms of Oslo's future success.

I’ll take this opportunity to share an initial attempt I made earlier this year to relate, rather broadly, the core ‘M’ language artefacts to multi-level metamodel architecture and also to the ‘syntax-semantics-pragmatics’ trinity.    This diagram is entirely of my own making and is in no way ‘official’ (I don’t work for Microsoft).   One major weakness is that it suggests a highly repository-centric emphasis in regards to pragmatics.   In reality, I expect that the use of the repository will only be one of many aspects of pragmatics.  Consider, for example, the role of MGraph is enabling pragmatics at the M0 layer or, indeed, the use of ‘Quadrant’.    This diagram is very much a work in progress and I will wait until future iterations before revisiting and refining this.   Another issue is that, on the left-hand side, I have suggested the existence of MSchema and MGrammar grammars.   These may not necessarily be made public by Microsoft, but do appear to exist.

Oirst we saw the great shake-out in the rules processing world - ILog went to IBM, RuleBurst/Haley went to Oracle, etc.   Now consolidation and merger raising its head in the younger, but related, world of CEP (Complex Event Processing) with the coming together of two leaders in that field, Aleri and Coral8.   We are probably still some way off seeing most of the main CEP playersl snapped up by the biggest names in the industry, but it may happen - IBM already bought AptSoft.

Both the rules companies mentioned above are members of Microsoft’s Business Processing Alliance; it is interesting to note that Microsoft is now effectively in alliance with two of their direct competitors in the BPM space.   Currently, however, CEP remains unknown territory to most .NET developers.   Will there be a major CEP story on the Microsoft platform in future?   We will have to wait to see.

Best wishes to the new Aleri

UPDATE:  It would appear, from the Microsoft web site, that ILOG (IBM) and RuleBurst (Oracle) have dropped out of Microsoft's Buisness Process Alliance.   They are no longer listed as members.

As promised in my last post (http://geekswithblogs.net/cyoung/archive/2009/02/24/129639.aspx), I had a look at the Express edition of Microsoft Solver Framework (http://code.msdn.microsoft.com/solverfoundation), and used it to implement the Einstein puzzle using the built-in CSP solver and C#.  Creating the solution proved particularly easy for the simple reason that Microsoft provides a worked example of a politically corrected ‘Zebra’ variant in their documentation (no tobacco or beer in sight).  You will find this as Sample 8 in the SFS Programming Primer document that comes with the download.

To read more and see the code, visit http://geekswithblogs.net/cyoung/archive/2009/02/25/129672.aspx

A few days ago, Daniel Selman of ILOG (now owned by IBM) published a solution to the Einstein puzzle.   See http://blogs.ilog.com/brms/2009/02/16/einsteins-puzzle/.   He did this in response to a challenge from James Owen who is one of the organisers of the October Rules Fest conference.  James invited the various vendors who have involvement in the conference to provide a solution using whatever approach they deemed best.   See http://orf2009.blogspot.com/2009/02/puzzle-1-for-conference.html.

Daniel suggests that, if ILOG had used a Rete rules engine, they would have had to implement 'convoluted rules' to solve the problem. Daniel is correct, but I thought it would be interesting to look a little deeper at why this is the case.

Read more at http://geekswithblogs.net/cyoung/archive/2009/02/24/129639.aspx.

The latest CTP version of the Oslo SDK was released a few days ago.    I spent some time yesterday installing it and having a look.    From a functionality perspective, there is not very much difference to the October 2008 CTP released after last year's PDC.    However, if you open up the main assemblies in Reflector, it quickly becomes apparent that the code has undergone significant refactoring and improvement.    The code looks much tidier and closer to production quality.   For example, the October 2008 CTP contained generated parser/lexer code for MGrammar.   In the latest version, this has been removed, and it appears that the parser code is now dynamically generated at runtime (the 'preferred' Oslo approach).   There has been lots of tidying up done in terms of type and method names.   Additional functionality has been added to manage various issues, and the entire code base looks tighter and better constructed. 

In terms of new functionality, this has been discussed elsewhere.   See, for example, http://www.alexthissen.nl/blogs/main/archive/2009/01/31/improvements-and-changes-to-oslo-sdk-and-repository-in-january-ctp.aspx.   Most attention has been given to the ability to include actions on the RHS of token productions.   In the work we have undertaken to date, we have come across at least one situation which requires this new feature, and which we could not properly address in the October 2008 CTP. 

Perhaps the most intriguing aspect of the January 2009 CTP is the way this new feature is described in the Release Notes.   I suspect that Microsoft has inadvertently let slip a feature that they didn't mean to go public on.   The Release Notes published on the web site state that: 

"Any production in a token can now have a code action or a graph action (formerly known as term construction)!  You can now specify a return type for a token definition in the case of code actions, similar to a syntax definition."

In the October 2008 CTP, actions are limited only to MGraph expressions.    An action is an optional implication of a production that controls the output of the MGraph abstract syntax tree created by the parser.   As far as I can tell, this is still the case in the new CTP.  Unlike many similar technologies, the CTP version of MGrammar does not support the inclusion of code statements as semantic actions.   This was discussed by Clemens Szyperski (an Oslo architect) in a comment to the blog article at http://weblogs.asp.net/fbouma/archive/2008/11/05/designing-a-language-is-hard-and-m-won-t-change-that.aspx, and the suggestion appears to be that this is a deliberate strategy in order to ensure that MGrammar remains (relatively) simple to write and focussed only on composable DSL creation.

The Release Notes statement suggests that Microsoft is looking at including code actions in MGrammar.   As I say, it would appear that this feature is not actually supported in the January 2009 CTP.   If it is, there is certainly no documentation explaining how to use this feature.   Interestingly, this may be the explanation for a feature within the October 2008 CTP which seems to have disappeared from the current version.   In the previous CTP, the MGrammar assembly included code for parsing C# statements.    The parser didn't appear to be designed as a full-blown C# parser, but looked like it was designed to parse code statements and expressions.    This code appears to be missing from the January 2009 CTP.

It is generally a fool’s errand to speculate on what is happening behind the scenes, and I certainly have no special insight or knowledge about Microsoft's intentions.   However, I can't help wondering if Microsoft has accidently let us see that they are considering supporting code actions in MGrammar when it is released, and have built code to support this feature which they do not wish to make public at the current time.   If this is the case, there is no guarantee that this feature will make it into the final release.   For my part, I have been thinking about this issue for a few months now, and am undecided, myself, as to the desirability of supporting full-blown semantic actions in this fashion.  The issue, I think, is about how useful this feature will really be in mainstream DSL creation.    Does the reduced problem domain of a domain-specific language imply language simplicity that generally avoids the need to handle complex semantics at the parser level?    Clearly, there is no fundamental reason why a DSL should not exhibit such complexity, but if the vast majority of DSLs are inherently simple, maybe it would be wiser to stick to the current labelled graph-only model employed by MGrammar parsers, and work around this restriction.   As I say, I am undecided.   It may be that the Oslo team are currently also uncertain of the best strategy.   It would be interesting to hear views from the wider modelling community.

One of the best ways to understand the concepts that lie behind a new technology is to compare and contrast it with similar technologies provided by other communities.   ‘Oslo’ addresses the modelling domain which has enjoyed a great deal of attention over several years.    This article is an attempt to articulate something of the nature of Oslo by relating it to the wider world of modelling, especially as envisaged by the OMG (Object Management Group).  

http://geekswithblogs.net/cyoung/archive/2009/01/05/128369.aspx

I spent some time today looking at Microsoft's Enterprise Service Bus (ESB) Guidance Toolkit, and this reminded me of an issue which we noticed a year ago, and did attempt to raise with Microsoft at the time, but which I never blogged about.   It's always a little uncomfortable taking the good people at Redmond to task, but, while this is hardly the most urgent problem facing today's world, it is, nevertheless, a bit of a problem and something which should be addressed.

In the article, I discuss the problem, where the Toolkit goes wrong, how UDDI should be used, and a possible remedy.    The article is at http://geekswithblogs.net/cyoung/archive/2008/11/12/126975.aspx.

I'm delighted to report that Microsoft is currently asking for responses to an on-line survey on the use of Microsoft BRE - delighted because it indicates that serious thought is being given to the evolution of this technology.    If your organisation uses MS BRE, do please take a few minutes to provide feedback.

The survey is at:   https://live.datstat.com/MSCSD-Collector/Survey.ashx?Name=BRE_Usage_Survey_Blog

The only issue I would highlight is that the survey deals mainly with present and past use of MS BRE, and not about how its use could be extended in the future.

The last couple of weeks have seen a significant increase in terms of announcements and information from Microsoft ahead of the Professional Developer’s Conference next week.   It is a key time for Microsoft’s Connected Systems Division (CSD) as they go public with their plans for .NET 4, Oslo and ‘Dublin’.  Microsoft’s announcement of Dublin led immediately to an interesting, if somewhat predictable, debate within the company I work for, and I think it is worth going over some of that territory here.

http://geekswithblogs.net/cyoung/archive/2008/10/15/125848.aspx

Microsoft announced BizTalk Server 2009 today, and gave the green light to talking about the new version.    It’s due for release in the first half of next year, and is shaping up nicely.    Microsoft is casting BizTalk Server 2009 as a major new version in its own right, rather than just an updated 'release' of BizTalk Server 2006.   This is an important move, and one I strongly welcome.   There is certainly enough in BizTalk Server 2009 to warrant thinking of it as a major revision of the product, although it retains the same familiar functionality and tooling we have been using since 2006 (or even 2004).

I've been fortunate in getting my hands on the current non-public CTP in the last month or so, and putting aspects of the new version through its paces.   It's not wise to go into much detail about this first CTP because some of those details will doubtless change in forthcoming betas, driven in part from the feedback Microsoft is getting.   However, I will say a little about the new development and build features and expand just a little on the press releases.   These are the areas I have been looking at in depth.

BizTalk Server 2009 will ship with bindings for Visual Studio 2008, and you will need to upgrade to this version of the IDE if you are still using Visual Studio 2005 (which you will be, of course, if you are a BizTalk Server 2006 user).  It happens that, in previous months, I've seen more than one BizTalk shop where developers are having to run both versions of the IDE side by side in order to continue developing for BizTalk Server 2006 whilst exploiting features like WCF and WF in .NET 3.5.   For many, the changes in BizTalk Server 2009 will come as a relief.   This is not a cynical ploy by Microsoft to force people into upgrades for no additional benefit.   The move to Visual Studio 2008 is accompanied by a very welcome move to a new project format, bringing BizTalk Server 2009 into line with mainstream .NET development.   The new BizTalk project type is defined using MS-Build, just like C# or VB.NET projects.   This has major implications in a number of areas.   First, it means that, if you use TFS Build, you can now build BizTalk Server 2009 projects without having to write complex scripts that shell out to DevEnv.    Just like C# projects, you can let TFS Build do the majority of the grunt work for you, and concentrate your attention more fully on ensuring that your automated build scripts are comprehensive and robust.   This is worth the upgrade in its own right, and removes a major source of current irritation.   Thanks to the rough edges in the current CTP, one of my colleagues has had the opportunity to get to grips with this side of BizTalk Server 2009 in some depth, and we can report that it is all looking very good indeed, once a few remaining gremlins have been chased out.

The other aspect of BizTalk Server 2009 which we have spent some time on is unit testing and debugging.   In one sense, this is not really so much about new functionality, but more about bringing what has previously been considered 'black-belt' into the mainstream.    In Biztalk Server 2006, we currently use BizUnit to drive black-box (grey-box?), end-to-end testing, but we have also created some additional code of our own to support unit testing of BizTalk maps, schemas and pipelines.    Similarly, if you know the undocumented registry key in BizTalk Server 2006, you can get BizTalk to retain the generated C# code for debugging purposes, manually attaching to BtsNtSvc.exe processes in order to debug into orchestrations, etc.    This has proved a life saver in certain situations.  The new version of BizTalk Server is now designed to support these approaches seamlessly.   Of course, it is designed to work with the integrated unit testing features of the IDE, rather than BizUnit.   There are currently some outstanding questions with regard to the debugging support which I won't go into here, because they simply reflect the unfinished state of the first CTP.   However, unit testing certainly works smoothly with BizTalk Server 2009 projects, and will help to raise the bar in terms of the approach that is taken to development of BizTalk solutions.   Again, I will avoid going into further here because I will hit issues that are still to be fully resolved, but things are generally looking good.

I have only mentioned those areas of BizTalk Server 2009 which I have been looking at in any depth.   There is a lot, lot more.   One thing I will monitor closely is the inclusion of ESB Guidance 2.0.   I have some problems with some aspects of ESB Guidance 1.0, like the unfortunate way in which the UDDI resolver violates the UDDI standard!    However, having spent the last year doing little else but designing and implementing service bus patterns using BizTalk Server and WCF (or, in one case, WSE), I regard the inclusion of ESB Guidance 2.0 as a really intriguing, and hopefully worthwhile, aspect of the new version.    It is also intriguing that Microsoft has decided to release their implementation of UDDI 3.0 as part of the BizTalk package.   Let no one tell you that BizTalk has no role to play in building service buses, or that it is to be regarded as 'merely a hub-and-spoke message broker'.   In my experience, the exploitation of the dynamic features of BizTalk (dynamic ports, BAM interception, the rules engine, etc.,) in a rigorous, policy-driven fashion provides good support for implementing many of the core patterns described within the world of ESB.   The core design of BizTalk Server predates later ESB thinking, but is much better aligned to it that some people will admit.

Roll on 2009.

Jurgen Willis, who is Group Program manager of Microsoft's Connected Framework Team over in Seattle, has been in touch, and I offered to forward his request on via this blog site.   Microsoft is currently looking for people to work in their rules area (WF Rules, etc).   They are specifically looking for a Program Manager and a Development Lead

• Community PM
• Activities PM
• Runtime PM