Charles Young

Looks interesting. Should be interesting to see how it evolves. I wonder how far they've delved into the general problem of aggregations and modifications. I skimmed the paper real quick and they mention retracting events, which implies they support modifications.

I'll be there and try my best to summarize tomorrow evening.

Excellent. Thanks Richard.

Hi Charles,

I'll be surprised (not the first time), if Microsoft is stupid enough to call a stream processing technology, CEP.

There is nothing in David Luckham's original papers on CEP where the state-of-the-art of complex detection oriented problems is "solved" by a stream processing engine.

Like I said, if they do, it will not be the first time, but I find it hard to belive that Microsoft would do this.

Yours faithfully, Tim

Well, to be pedantic, any CEP engine is going to have to be able to process event streams in order to detect patterns across those streams. However, a major characteristic of Microsoft's CEDR technology (on which I understand the engine is based) is that it supports speculative processing in order to handle a key characteristic of the event cloud is an efficient manner - specifically, that events may be delivered out-of-order in terms of timestamps or causality. A CEP engine ought to be able to handle such 'degraded' streams as well as situations where correlation between different events in different streams is not determined in a nice neat sequential fashion

These features take CEDR well beyond the most basic concept of event stream processing and should make a significant contribution to realising the concept of CEP, especially in the context of all those analytics capabilities that are already bundled with SQL Server.

Hi Charles,

To be fair to the rest of the world of technology and decades of prior art, there are many, many papers, concepts, people and applications in the world related to processing events well beyond what is described in CEDR. Furthremore, the group of good folks and others, who seemingly focus on rules (often more accurately query engines) and the application of these query engines to stream-oriented processing, again are seemingly keen to identify any technology that uses gueries (or in some cased rules) to process a stream of events, as "CEP".

One observation is that in the papers you site, Microsoft does not use the term CEP, not once (I pedantically did a search on the PDF file, LOL). Microsoft was, in my view, correct, in being careful not to write a paper based on a marketing phase with little tangible technical context or definition, and to mostly use terms like "stream processing".

After all, in David Luckham's timeline, in the beginning the good professor referred to CEP as a technology to detect (complex events) information from message-based distributed systems (mostly network security and distributed system simulation/debugging) and to deal with problems in cauality, not scheduling or routing based on stream context.

Then, the good professor and team redefined CEP to be a "software analyst" style term, to meld with BAM, SOA, EDA, and so forth and so on, so to have such an abstract meaning as the term "CEP" can be anything with logic inside an software engine!

However, in practice vendors using CEP are mostly query engines that process streaming data in a few niche markets, and the rest of the world of folks processing complex events have ignored the term, as it has little to no impact on their work, which is generally more advanced, as Peter Lin et al often agrees.

In other words, you and others make excellent points, but are seemingly applying those points to a narrow group of concepts, streaming products and vendors and not the larger world of folks who have been doing similar things for decades, and who don't call what they do "CEP". I know there is no malice in this association, but the field of processing complex event is significantly larger and more interesting that what Microsoft as discussed in CEDR. Why not talk about what is happening in air traffic control? Complex detection in cybersecurity using ANNs? Why machine learning is required for most classes of adaptive complex problems? Why Bayesian analytics dominate must current classification problems, real-time or not?

What I am getting to is a core problem with "CEP thinking", and a group of us continue to try to get the smaller "CEP thinking" "stream processing community" to embrace the prior art and move beyond the not-invented-here world view that is pervasive in that community.

By definition now, that community has defined CEP as any processing that combines two or more events into another event ... for example, according to the EPTS/Luckham/et al definition of "complex event", the event handler tracking your mouse at your computer is "CEP".

In practice, of course, "CEP" is stream processing query engines. Maybe by design "CEP" has such a broad defintion as to be "all things to all people" and narrow enough in practice to be "all things to all people" by a handful of software vendors, LOL.

Anyway, I digress :-) Lucky for me, my thinking is not inspired or influnced by the need to satisfy the increasing hunger of my venture capital angels or my marketing team.

Pedantically Yours,

Tim

Well, as it happens, for some time my line in the rules community has been that CEP may be a new(ish) term, but sits on top of decades of effort with regard to understanding event processing, including a fair share of involvement within the rules community. So, I have sympathy with what you say. TLAs come and go and are just part of the background noise of this industry. Real evolution is a less effervescent affair which generally builds incrementally on what has gone before and establishes new connections between existing capabilities.

As far as Microsoft's announcements are concerned, the association between CEP and SQL Server is critical because it links their engine not only with data management but with their main platform locus for analytical technologies - so, for example, MS already bundles their Analysis Server with SQL Server, and this, in turn, delivers various implementations for Naive Bayes, neural network, time-series/regression and other analytical algorithms. From my BPM/Rules perspective, another very interesting point is that, after recent re-orgs, the BizTalk Server team in Redmond is now closely aligned with the SQL Server team. I have already heard lots of speculation about possible tie-ups between the CEDR engine and, for example, the RFID Server which ships with BizTalk and which effectively provide sensor network management. So, I would say that Microsoft’s strategy appears to be one of aligning a number of complementary competencies and capabilities in order to deliver a much more holistic approach to CEP and EDA in the future.

It’s a process of evolution, and the new engine will play its part in that evolution.