Geeks With Blogs

Charles Young

At first sight, StreamInsight doesn’t appear to offer any support for dynamic queries.   Once you have got over the initial ‘wow’ factor of creating continuous queries over streams using LINQ, rather than some SQL dialect or specialised DSL, the reality hits you that your query is just code.   You wrote the LINQ using C# or VB.NET, and you ended up compiling it into some assembly.

That’s all very well, but it’s not very flexible.   How are you going to store and manage your queries in a repository so that you can review, change and version them?   Does Microsoft really expect you to recompile and re-deploy assemblies every time you want to create or change a query in StreamInsight?   Surely you should be able to dynamically deploy queries straight from a repository.

Have no fear.   All is well.   The QueryTemplate is serialisable.   It isn’t serialisable in the .NET sense (i.e., it isn’t marked up as serilaisable, and it doesn’t implement ISerializable).   Rather, it implements the QueryTemplate.Definition property which returns an XML document containing a complete rendering of your template, complete with a full dataflow and all the operators.   To deserialise  the XML, you use Application.CreateQueryTemplate.   You have to pass in an XmlReader that you have created over the XML.   You get back a clone of the original QueryTemplate which you can now bind to input and output adapters and emit to the engine as a Query.   Nothing could be simpler.

There are a couple of gotchas.   First, don’t do what I did while experimenting.   I created a QueryTemplate and serialised it to XML.   I then tried to deserialise it as a second, identical, QueryTemplate in the same Server instance.    I got back an InvalidDefinition error.   It seems you can’t have identical QueryTemplates in a single StreamInsight Server instance.

The second thing to note is that the serialised QueryTemplate does not contain fully qualified references to your event types.   You still have to define the event types in your application by calling the CreateEventType method.   If you plan to store QueryTemplate XML in some custom repository, you will also need to store the fully qualified names of the .NET types your have defined for your events.   Of course, you will also need to ensure that these types are available at runtime.

Much the same argument applies to adapters.   The QueryTemplate XML does not contain any information on which input and output adapters you want to bind to your query.   However, once you realise that your QueryTemplates are serialisable, it is simple to work out how you might store and manage the template in some repository together with event type and adapter information.   All this information, taken together, defines the StreamInsight concept of a query.   You can write code to extract it from the repository at run-time and run the query.

The current CTP is just a glorified SDK.   Will Microsoft provide a repository and tooling in the release version?   I have no idea.   Given the freedom StreamInsight offers in terms of event definition, and given the fact that the engine can be hosted in custom applications, it is difficult to see how Microsoft could provide a totally generic end-to-end solution that would support externalisation of queries in any and all circumstances.   They could provide a repository and API, and leave it to developers to write the code to exploit the repository.   However, you would still need to write the LINQ together with code to create QueryTemplates and extract and upload XML to the repository.

Posted on Monday, September 28, 2009 7:26 PM | Back to top

Comments on this post: Serialising Microsoft StreamInsight QueryTemplates

# re: Serialising Microsoft StreamInsight QueryTemplates
Requesting Gravatar...
This is still an area where I have yet to make up my mind. I need to apply StreamInsight's LINQ approach to a real-world problem with sufficient complexity. Obviously, at one level, you couldn't really hope for a more expressive event query language - you have the full power of a GPL at your disposal, after all. At another, I keep on thinking...yes, but how do I express sequence?

The fact that a QueryTemplate serialises/deserialises to/from a complete dataflow definition in XML makes it easy to see you how you could implment non-LINQ approaches over StreamInsight.
Left by Charles Young on Sep 28, 2009 11:49 PM

# re: Serialising Microsoft StreamInsight QueryTemplates
Requesting Gravatar...
Honestly, I doubt the answer to the event processing language will be solved any time soon. The event processing society has been talking about it for over a year now.

From my survey of the existing literature on temporal logic, modal logic and FOL, the challenge is monumental. It's definitely going to take a long time to get a decent general purpose event processing langauge.
Left by Peter Lin on Sep 30, 2009 11:13 AM

# re: Serialising Microsoft StreamInsight QueryTemplates
Requesting Gravatar...
It is totally sad that the StreamInsight team decided to roll their own XML serialization, instead of using WCF. And since most will be .Net clients, not sure why they decided to go XML over a nice binary serialization. The XML they are sending is ugly, and without any schema or documentation, so interop isn't really possible.

That being said, LINQ Expressions are not serializable, so you have to roll your own to serialze them. But, the ADODS (aka Astoria) team as already done a LINQ Expression to URL serialization. I'm sure they could have used most of that (if not all).

And don't even get me started on the threading model in StreamInsight. .Net 4.0 would have helped.
Left by Don Demsak on Sep 30, 2009 3:47 PM

# re: Serialising Microsoft StreamInsight QueryTemplates
Requesting Gravatar...
Good point about LINQ expressions not being serialisable. Before I discovered QueryTemplate.Definition, I went looking to see if you could get an expression tree from a CepStream (like you can with IQueryable). Answer? No! Nothing doing.

Obviously this is just the first public CTP, so it is not too surprising that we don't have a schema or documentation for the XML yet. After all, they haven't necessarily finalised the shape of the XML. Schemas and documentation will hopefully be there by the time they've finished productising the technology.

One good thing about using XML over binary serialisation is that it may help foster the creation of custom DSLs and language translators over StreamInsight.

I like your idea of using WCF. I guess you have in mind exposing a WCF data contract for a QueryTemplate. Luckham's concept of an EPN fits naturally with ESB thinking. I'd like to see the ability to deploy QueryTemplates dynamically from a central repository across a distributed environment to my StreamInsight agents. So, better support for WCF would be welcome.
Left by Charles Young on Sep 30, 2009 4:46 PM

# re: Serialising Microsoft StreamInsight QueryTemplates
Requesting Gravatar...

Do you have any working example that shows how to deserialize a QueryTemplate from XML? I've tried to do it, but I didn't succeeded... :-(


Left by Klaus Aschenbrenner on Nov 16, 2010 6:42 PM

Your comment:
 (will show your gravatar)

Copyright © Charles Young | Powered by: