Charles Young

Sunday, August 29, 2010 #

Oh my. Somehow I missed this one from earlier this month. A guy from HP Labs called Vinay Deolalikar claims he has the proof. P ≠ NP. I'm nowhere near clever enough to follow his arguments (see http://www.hpl.hp.com/personal/Vinay_Deolalikar/Papers/pnp_synopsis.pdf for a synopsis), but I understand the impact.

If he is right, then in a sense, nothing changes. It's just that we will now know for sure that there is no magic formula that will revolutionise the efficiency of our software. The proof will confirm what has been long suspected. However, if the proof holds up, then it is a major milestone in computer science. I mean MAJOR.

Nothing personal against Mr Deolalikar, but I really hope the proof does not hold up. The idea that P could just possibly equal NP is so beguiling.

No idea what I am talking about? Here comes Wikipedia to the rescue! http://en.wikipedia.org/wiki/P_%3D_NP_problem

And here is a really easily digestible explanation from the good ol’ BBC. Jigsaws work for me!

http://www.bbc.co.uk/news/science-environment-10938302?utm_source=SamacharNepal&utm_medium=SamacharNepal.

posted @ Sunday, August 29, 2010 1:14 PM | Feedback (5)

Saturday, August 28, 2010 #

IE9 Chrome

It was always obvious that IE9 was going to get new chrome. Why else publish the technical previews with a mocked-up UI? The very fact that the IE team has cleanly separated the core browser technology from the old IE chrome indicates that they are free to change the whole look and feel of the browser if they wish. And of course, they must want to do so. Everyone loves the clean, zippy feel of Google's Chrome. If IE9 is to halt the slow slide in 'market share' that previous versions of the browser have experienced in recent years, it is Google they must take on, more so than any other browser. Yes, Firefox holds second place, but its growth has stagnated. Google Chrome, on the other hand, is on a steady and solid trajectory upwards.

After years of effort having to chase the alternatives and constantly being behind the curve, IE9 represents a real change in the fortunes of what is still the world's most widely used browser. Microsoft return to where they could always have been if they hadn't decided all those years ago to abandon further development of IE, loosing years of advantage in the process. IE9 has already forced Google and Firefox to speed up implementation of GPU support (it would be unthinkable for them not to have responded to the amazing graphic performance of Microsoft's preview). It is in grave danger of coming out top in just about every official standards compliance test. It's new Javascript engine is right up there, performance wise, with Google Chrome, Safari and Opera, and way ahead of Firefox. It will even shortly achieve 100/100 on Acid 3 (almost there now). And now we get our first glimpse of the new Chrome-like chrome. It's the dawn of a new age for the old browser and I can't wait to get my hands on the beta.

http://www.zdnet.com/blog/microsoft/is-this-microsofts-new-internet-explorer-9-interface/7213?tag=mantle_skin;content

posted @ Saturday, August 28, 2010 7:04 PM | Feedback (2)

Wednesday, August 25, 2010 #

See you at Rules Fest

I've been taking quite a prolonged break from blogging, partly because my attention has been elsewhere. Writing several chapters for BizTalk Server 2010 Unleashed (due to be published towards the end of the year) has taken up a lot of time. I'm also involved in helping or organise Rules Fest 2010. And then there is the day job,...and four kids,...and two cats,...and management of a local youth club,...and organising my wife's 50th, etc., etc. Anyway, this is an unashamed plug for the conference.

Rules Fest, (previously October Rules Fest) is in its third year, and has switched venue from Dallas to San Jose in the heart of Silicon Valley. It aims, simply, to be three days to solid wall-to-wall technical content for everything to do with rules, reasoning and inference. We aim the conference at the development community, although there is plenty of content for solution and enterprise architects, decision makers, etc. However, we are all about fostering and nurturing the real-world application of rules-based technologies by technical practitioners.

In previous years, we heard a fair amount about the implementation or rule engines. We realised that, appealing as this is to some, most of us are not involved in building rule engines and constraint handlers. We want to concentrate more on practical application of rules to real-world problems. However, Rules Fest continues to attract some of the best known names in the industry, including people who have been at the forefront of designing and implementing core rules technology for many years. Come, mingle with the stars.

This year, we have Professor John Laird as our keynote speaker. John is at the forefront of research into machine cognition. We are unashamedly geeks, and what he does is really exciting. After a foray into AI, we come down to earth with a range of presentations that should appeal to a wide audience. There is plenty on real-world knowledge engineering, design and implementation of rules-based systems, the impact of new standards such as W3C RIF, complex event processing, software development practices and methodology and a number of interesting case studies. Our speakers are experienced practitioners and experts with lots of wisdom to impart. If you are involved in rule-based development, or considering the impact of rules on your company or career, this is the conference for you.

The main part of the conference takes place between Monday 11th to Wednesday 13th October. Stick around for Thursday 14th. This is 'bootcamp' day with practical sessions for developers using a variety of mainstream technologies. The web site has more details.

For BizTalk readers, why not consider attending? This is a industry-wide conference that draws from a much wider world than some of us normally inhabit (shame on us). There is a lot to learn. If you are a BizTalk developer with a focus or interest in rules development, and want to attend, book on the web site, but also let me know via this site. If there is enough call for it, I could easily be persuaded to organise a bootcamp just for you!

Oh, and the conference venue is gorgeous, apparently. Don't tell your boss.

The web site is at http://rulesfest.org.
Follow us on Twitter @Rules_Fest_2010
Write on our FaceBook wall at http://www.facebook.com/pages/Rules-Fest-2010/108532835869540
Visit us at LinkedIn at http://www.linkedin.com/groups?gid=3334438

posted @ Wednesday, August 25, 2010 3:49 PM | Feedback (0)

Monday, August 23, 2010 #

VirtualBox and 100% CPU usage - the SuperFetch effect

I have a Dell laptop with 8GB RAM and a dual core processor running Windows 7. I use my machine quite intensively. Quite often I run virtual images under Oracle's VirtualBox. I'm currently using an image that contains an installation of the BizTalk Server 2010 beta.

For months, I found that every time I fired up images under VirtualBox, I would run into problems. I might be fine for a while, but then, at some point, problems would arise. The machine would become totally unresponsive for a few minutes. Task manager would report 100% CPU usage. After a while, things would revert to normal. A few minutes later, I would have to endure another period of unresponsiveness. This would continue for an hour or two, and then everything would revert to normal.

The same problem arose on my machine every Wednesday lunchtime, regardless of the use of VirtualBox. My machine is a company laptop. It has Forefront installed and scheduled to run a full scan at this time. I can't change the schedule because of policies applied to the machine, although our systems people assure me they have been careful to ensure that Forefront excludes very large files such as virtual images from the scan.

I've lived with this issue for a long time. Normally, I would hope to be able to spot a rogue Windows process that is hogging all the processor cycles. However, in this case, there was never any indication of any process causing problems. The combined count of all CPU usage of all processes, including processes belonging to other users, was always substantially below 100%. This was true for processes on the host as well as those running under VirtualBox. Another strange thing was that, over many weeks, I kept on seeing the emergence of a pattern (not counting the Wednesday lunchtime issue, which had an obvious link to Forefront). I might have three days when the problem occurred at the same time each day. Then the pattern would always disappear and some different pattern would later emerge.

Of course, I used various tools to try to track down the problem. With no rogue processes, my best guess was that the problem must be occurring at a lower level, perhaps with some badly behaving driver. I found no hint of any problems with drivers, though. I've seen rather similar behaviour in the past due to interrupt conflicts, but again, no sign of any issues. I spent quite some time with SysInternals Process Explorer trying to track down the problem, but to no avail.

Then, a few days ago, I got my first solid clue as to what was happening. It was Process Explorer that helped. My attention was drawn to an instance of the Windows service host that I could see was using up a few % of CPU cycles. I opened it up and had a look. One of the nice things about Process Explorer is that it provides a graph of CPU usage at the process level. The graph I saw grabbed my full attention. There, before my eyes, was a lovely trace showing clearly that, at just the same time my machine had gone into 100% CPU usage, this process had suddenly started using a few % of CPU cycles. At the moment the CPU usage dropped back down, so did the graph.

Process Explorer allows you to see all the services that are running in an instance of the service host. I set things up and waited for the problem to re-emerge. Sure enough, after a minute or two, CPU usage rocketed sky high. I had previously discovered that if I paused VirtualBox, the CPU usage would drop back to about 80%. The machine was still very sluggish, but could be used. So, I paused VirtualBox, waiting an eternity for the mouse click event to be processed, and then got to work. As quickly as I could, I worked through the list of services. The host was running exactly ten Windows services. I stopped the Desktop Windows Management Session manager [UxSms] - bang went my Aero interface - I stopped the Distributed Link Tracking Client [TrkWks] service - no change - I killed the Human Interface Device Access [hidserv] - etc., etc. At last, on the seventh service, I stopped SuperFetch and, after taking ages to close, everything burst into life.

I have been running VirtualBox constantly since then, over several days. I have yet to see any reoccurrence of the issue. Last Wednesday, for the first time in a very long time, Forefront completed a full scan without issues. Wonderful.

Is SuperFetch at fault? I can't say. Is it just a bad installation of Windows 7. Maybe. Perhaps VirtualBox is the true culprit. That's possible. I have no idea. All I know is that my productivity is now significantly higher after switching SuperFetch off. To switch it off, I simply opened the 'Services' administration managament console and disabled the SuperFetch Windows service.

I discussed my experience on the BizTalk Gurus newsgroup and two other people responded that they had been having similar issues. They have both switched SuperFetch off. One, Randal van Splunteren, got back to me to say that his VirtualBox problems were significantly reduced by switching SuperFetch off. However, as he pointed out, there was still a fairly high on-going CPU usage when VirtualBox is running (45-50% on my box). As I understand it, VirtualBox always soaks up CPU cycles, even when the image is idle, due to timing interrupts which I presume has something to do with synchronising the virtual image to the actual hardware. This is to be expected, but the CPU usage did seem too high for comfort. Randal investigated further and came up with a further improvement. Both he and I had configured our images to use two logical CPUs. You can control this on the Processor tab under Settings/System for a specific image. Reducing this to 1 significantly reduced the CPU usage.

Randal reports that this only really helped once he had reverted back to an older version of VirtualBox. Under Oracle's ownership, we are currently at version 3.2.8. Randal recommends ditching this version in favour of version 3.1.8 which belongs to the Sun era. I experimented on my machine with both versions. My experience was that setting the number of logical processors to 1 made a significant different on both versions, but the effect was a little greater under the older version. I get about 20% CPU usage under 3.2.8, but perhaps only 15% under 3.1.8. Randal has decided to use the older version. I've decided to stick with the current version, at least for now.

So, a combination of disabling SuperFetch and configuring a single logical processor has made all the difference. If you have also been having problems running VirtualBox, then there may also be merit in reverting to an older version.

posted @ Monday, August 23, 2010 9:20 AM | Feedback (0)

Saturday, March 06, 2010 #

Form, function and complexity in rule processing

Tim Bass posted on ‘Orwellian Event Processing’. I was involved in a heated exchange in the comments, and he has more recently published a post entitled ‘Disadvantages of Rule-Based Systems (Part 1)’. Whatever the rights and wrongs of our exchange, it clearly failed to generate any agreement or understanding of our different positions. I don't particularly want to promote further argument of that kind, but I do want to take the opportunity of offering a different perspective on rule-processing and an explanation of my comments.

For me, the ‘red rag’ lay in Tim’s claim that “...rules alone are highly inefficient for most classes of (not simple) problems” and a later paragraph that appears to equate the simplicity of form (‘IF-THEN-ELSE’) with simplicity of function. It is not the first time Tim has expressed these views and not the first time I have responded to his assertions. Indeed, Tim has a long history of commenting on the subject of complex event processing (CEP) and, less often, rule processing in ‘robust’ terms, often asserting that very many other people’s opinions on this subject are mistaken. In turn, I am of the opinion that, certainly in terms of rule processing, which is an area in which I have a specific interest and knowledge, he is often mistaken.

There is no simple answer to the fundamental question ‘what is a rule?’ We use the word in a very fluid fashion in English. Likewise, the term ‘rule processing’, as used widely in IT, is equally difficult to define simplistically. The best way to envisage the term is as a ‘centre of gravity’ within a wider domain. That domain contains many other ‘centres of gravity’, including CEP, statistical analytics, neural networks, natural language processing and so much more. Whole communities tend to gravitate towards and build themselves around some of these centres.

The term 'rule processing' is associated with many different technology types, various software products, different architectural patterns, the functional capability of many applications and services, etc. There is considerable variation amongst these different technologies, techniques and products. Very broadly, a common theme is their ability to manage certain types of processing and problem solving through declarative, or semi-declarative, statements of propositional logic bound to action-based consequences. It is generally important to be able to decouple these statements from other parts of an overall system or architecture so that they can be managed and deployed independently.

As a centre of gravity, ‘rule processing’ is no island. It exists in the context of a domain of discourse that is, itself, highly interconnected and continuous. Rule processing does not, for example, exist in splendid isolation to natural language processing. On the contrary, an on-going theme of rule processing is to find better ways to express rules in natural language and map these to executable forms. Rule processing does not exist in splendid isolation to CEP. On the contrary, an event processing agent can reasonably be considered as a rule engine (a theme in ‘Power of Events’ by David Luckham). Rule processing does not live in splendid isolation to statistical approaches such as Bayesian analytics. On the contrary, rule processing and statistical analytics are highly synergistic. Rule processing does not even live in splendid isolation to neural networks. For example, significant research has centred on finding ways to translate trained nets into explicit rule sets in order to support forms of validation and facilitate insight into the knowledge stored in those nets.

What about simplicity of form? Many rule processing technologies do indeed use a very simple form (‘If...Then’, ‘When...Do’, etc.) However, it is a fundamental mistake to equate simplicity of form with simplicity of function. It is absolutely mistaken to suggest that simplicity of form is a barrier to the efficient handling of complexity. There are countless real-world examples which serve to disprove that notion. Indeed, simplicity of form is often the key to handling complexity.

Does rule processing offer a ‘one size fits all’. No, of course not. No serious commentator suggests it does. Does the design and management of large knowledge bases, expressed as rules, become difficult? Yes, it can do, but that is true of any large knowledge base, regardless of the form in which knowledge is expressed.

The measure of complexity is not a function of rule set size or rule form. It tends to be correlated more strongly with the size of the ‘problem space’ (‘search space’) which is something quite different. Analysis of the problem space and the algorithms we use to search through that space are, of course, the very things we use to derive objective measures of the complexity of a given problem. This is basic computer science and common practice.

Sailing a Dreadnaught through the sea of information technology and lobbing shells at some of the islands we encounter along the way does no one any good. Building bridges and causeways between islands so that the inhabitants can collaborate in open discourse offers hope of real progress.

posted @ Saturday, March 06, 2010 1:39 PM | Feedback (10)

Tuesday, February 23, 2010 #

BizTalk Server: Handling decimal types with the xpath() Function

Last week, I spent some time looking at a problem with a customer's orchestration. In one place, the developer had made use of the infamous BizTalk xpath() function to assign a value in one message to a decimal field in another message. He got an error stating that the application is "Unable to cast object of type 'System.Double' to type 'System.Xml.XmlNodeList'". This is due to a logical error in BizTalk's code. Read more at http://geekswithblogs.net/cyoung/archive/2010/02/23/biztalk-server-handling-decimal-types-with-the-xpath-function.aspx.

posted @ Tuesday, February 23, 2010 3:17 PM | Feedback (0)

Sunday, January 31, 2010 #

Book plug: 'Intelligent Systems' by Robert Schalkoff

I am deeply engrossed at present in a new book, published at the beginning of this year, called 'Intelligent Systems - Principles, Paradigms and Pragmatics' by Robert Schalkoff, and published by Jones & Bartlett. Rather strangely, I note the copyright year is 2011, so I guess it fell through some kind of wormhole from the future. I don't think I have enjoyed reading an IT book as much in the last 20 years.

The book is a modern introduction to the whole field of 'intelligent systems' (IS) which is broadly what we used to call AI before the term fell out of favour. It is very much an introduction "suitable for a first course in IS...anywhere from the junior level undergraduate to first year graduate level". It is 700+ pages of excellence. I'm glad to say that I'm fairly familiar with the ground covered in the first few chapters, despite the absence of a degree in Computer Science. What I really like about it, though, is that it covers a wide range of subjects, is grounded in practical use of freely available tools (Protege, CLIPS, Soar, etc.,) and is written by someone who clearly understands what it is like to be unfamiliar with the subject and the kind of questions that beginners ask. It doesn't drown you with algebra, but uses enough to illustrate the points being made. It concentrates almost more on worked examples of CSPs, rule sets, blackboards, decision trees, fuzzy logic and all kinds of other stuff than it does on maths. An hey, Microsoft's BRE even gets a mention in dispatches on page 171 :-) I'm happy! I have some way to travel yet before I get to the chapters on neural networks, learning systems, genetic algorithms and the like, but I will enjoy the road.

I strongly recommend this book for anyone who is looking for a comprehensive and highly readable introductory overview of IS.

posted @ Sunday, January 31, 2010 7:06 AM | Feedback (4)

Saturday, January 30, 2010 #

Microsoft Field Surveys

So there I am doing what a geek does best - sitting in Cafe Nero on Tottenham Court Road in the centre of London on a Saturday afternoon, killing time while waiting to meet up with my daughter, and doing a little coding to pass the time when someone (his name turned out to be Jim) asks me "so what do you think of the Visual Studio 2010 beta?"

Me> It's OK; its stable and functional, and the UI performs quite well, which is good because it uses WPF.

Jim> Yes I know, I'm working on it.

Me> (slightly confused, and thinking he means he is working with WPF) Oh, that's interesting. So what kind of code do you write?

Jim> I'm working on Visual Studio.

Me> Yes, I realise, but what are you using WPF for?

Jim> No, I mean I work for Microsoft. I'm in the Visual Studio team over in Redmond. I work on Visual Studio. Always good to get feedback!

Beats an MSDN survey any day of the week!

posted @ Saturday, January 30, 2010 2:09 PM | Feedback (1)

Monday, January 18, 2010 #

MS BRE Backward Chaining - Part 3: The ‘Bound Variable' pattern

This is the third and final part of a three-part series, as follows:

· Part 1: Situating the Business Rules Engine

· Part 2: Backward Chaining: the 'Simple Name' pattern

· Part 3: Backward Chaining: the ‘Bound Variable' pattern

In Part 2 we looked at how developers can use procedural attachments and 'directive events' to implement a simple form of backward chaining for Microsoft's Business Rule Engine. In this third, and final, part, we will extend the code to tackle more complex backward-chaining requirements. The 'simple name' goal pattern is fairly easy to understand, but is limited. It provides a coarse-grained mechanism for switching sets of rules 'on' and 'off' in a backward-chained fashion. When this model fits the problem, it can work well. However, many real-world problems requires a greater degree of expressivity.

posted @ Monday, January 18, 2010 5:29 AM | Feedback (0)

MS BRE Backward Chaining - Part 2: The 'Simple Name' pattern

This is the second of a three-part series, as follows:

In part one of this three-part series, I explained some of the historical influences that led Microsoft to design their Business Rules Engine as a 'situated reasoning engine'. In this second part, I want to look at a pattern that exploits these features. Microsoft's engine is often described as a 'forward-chaining' rule engine. However, like several other similar engines, it is entirely capable of undertaking 'backward-chaining' as well. It has no explicit, built-in support for the semantics of backward-chaining, which is a pity. However, the 'situated' capabilities of the engine are sufficient to implement the backward-chaining approach.

posted @ Monday, January 18, 2010 5:27 AM | Feedback (0)

MS BRE Backward Chaining - Part 1: Situating the Business Rules Engine

This is the first of a three-part series, as follows:

Karl posted on forward and backward chaining at http://karlreinsch.com/2010/01/07/on-chaining/. I emailed him privately and talked a bit about how backward chaining can be implemented on Microsoft's Business Rule Engine (I even sent him a little example). A couple of days later he published a second post at http://karlreinsch.com/2010/01/11/have-you-implemented-backward-chaining-on-a-microsoft-rule-engine/. So, I shall rise to the challenge and explain how backward chaining can be done in the BRE. However, I'll leave that for parts 2 and 3. In this first part, I want to go over some of the history regarding the relevant features of the BRE and why they are there.

posted @ Monday, January 18, 2010 5:25 AM | Feedback (0)

Thursday, December 17, 2009 #

5 minute interview

Well, here is a shameless piece of self-promotion. Five minutes with a Rules nut, thanks to Tibi Covaci from Cloudcasts.

http://www.cloudcasts.net/ViewWebcast.aspx?webcastid=2521412533822906559

One of the best things about working in a company is that the people you work with conspire to make sure you never get too big-headed. So here is an alternative video made by one of my esteemed colleagues...who I plan to sue for defamation!

http://elfyourself.jibjab.com/view/fY0TwwvDaeOq4nWi

NB, the second link expires on 15th Jan 2010

posted @ Thursday, December 17, 2009 6:26 AM | Feedback (0)

Friday, November 20, 2009 #

BizTalk Server 2009 - Rule deployment issue

There is a small configuration issue with the Microsoft Business Rules Composer in BizTalk Server 2009. This will only affect you if you deploy the rules engine by itself without the rest of BizTalk Server. In a full installation of BizTalk Server, BTS provides a set of BTS-specific Rule Framework components in as assembly called Microsoft.BizTalk.RuleEngineExtensions. This includes a RuleSetDeploymentDriver component that manages deployment of rules via the SQL Server repository.

The BTS-specific RuleSetDeploymentDriver has a dependency on WMI (Windows Management Instrumentation). In a full BTS installation, BTS creates an instance of a CIM class called MSBTS_GroupSetting and populates various property values from the BizTalk management database. This includes the assembly and class name of a RuleSetDeploymentDriver. If you install only the rule processing components, there is no BizTalk management database, and no instance of this class is created. Unfortunately, this means that, whenever you try to deploy a rule set from the Rules Composer, you get a rude message saying:

The database "<server>:<database>" associated with the deployment driver does not match the database ":" specified during product configuration

The message is entirely correct. Microsoft's BTS-specific RuleSetDeploymentDriver component performs this check presumably to ensure that registry settings for the rule repository database name and server name are identical to those configured in the BizTalk management database. This ensures that the Rule Engine Update Service is using the same BTS-specific RuleSetDeploymentDriver component.

When you install only the rule processing components, the registry is unfortunately configured as if a full BizTalk installation had been done. The Rule Composer tries to use the BTS-specific RuleSetDeploymentDriver component to deploy rule sets and fails because no instance of MSBTS_GroupSetting has been created in the CIM store.

Fortunately, this is a simple problem to fix. Microsoft provides a non BTS-specific RuleSetDeploymentDriver component for the rule repository database. All you need to do is change the registry settings to use this instead.

Open the Registry Editor (regedit) and locate the following key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\BusinessRules\3.0

If you have installed the rule processing components on a 64-bit version of Windows, they key will be at:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\BusinessRules\3.0

You should find the DeploymentDriverAssembly and DeploymentDriverClass values under the key. These should be changed to the following values, respectively:

Microsoft.RuleEngine, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
Microsoft.RuleEngine.RuleSetDeploymentDriver

That's it. The Rule Composer should now be able to deploy rule sets.

posted @ Friday, November 20, 2009 7:28 AM | Feedback (2)

Tuesday, November 17, 2009 #

SQL Server Modelling - November CTP

Hot on the heels of one CTP (StreamInsight) and timed to coincide with the opening day of the PDC, Microsoft has just released the latest version of the technology formally known as 'Oslo'. SQL Server Modelling, as we must now learn to call it, has several improvements over the previous CTP release last May. Indeed, the previous CTP was characterised by behind-the-scenes code improvements and rewrites rather than new functionality. It is therefore doubly reassuring to see the slew of new features.

I won't spend time going into new functionality here because Kraig Brockschmidt (yes, for all you 'techie's of a certain age'...this is the same Kraig B of 'Inside OLE' fame - every COM developer's bible circa 1995) has just posted an excellent summary at http://blogs.msdn.com/modelcitizen/archive/2009/11/17/announcing-the-sql-server-modeling-n-e-oslo-ctp-for-november-2009.aspx together with links to the download and various materials. In any case, it's getting late here and I need to go to bed. However, things are moving along. I'm looking forward to getting to grips with Quadrant which sounds like it may, at last, be beginning to make some sense. M improvements sound great.

Like others, I was deeply alarmed by Doug Purdy's post (see http://www.douglaspurdy.com/2009/11/10/from-oslo-to-sql-server-modeling/) a week ago, and my initial reaction was similar to many commentators (something along the lines of deep groans, punctuated by loud inarticulate screams). We must hope that the new name and product alignment, which appears at so many levels to indicate an abandonment of the deep platform vision many of us believed 'Oslo' represented, is just a temporary glitch in Microsoft's modelling story. SQL Server may be a platform. Most of us think of it as a product. Windows is the platform, and we want Microsoft to do better at supporting modelling across their entire platform, and not just one, admittedly very powerful, corner of their universe <end of rant>

posted @ Tuesday, November 17, 2009 5:20 PM | Feedback (0)

Microsoft StreamInsight CTP 3 - New Features

Microsoft has just released the November CTP (CTP 3) of StreamInsight. See http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=01c664e4-1c98-4fc8-93ee-08cc039503c1.

I've encountered some confusion from more than one person about the status of StreamInsight, so let me explain briefly that a CTP is a Community Technology Preview. It's not a beta as such. CTPs are closely aligned with the shorter iterative development cycles of agile methodologies. Many Microsoft product teams use them to provide insight into their progress from a much earlier stage than would be the case for a formal beta programme. So, StreamInsight is a work in progress. It hasn't shipped yet. It isn't necessarily 'code complete'. It doesn't have full documentation. Bits are missing or may not work. Under the terms of the license, you can't use it in production. For some reason, CTPs seem to cause a lot of confusion, even in the Microsoft community, but the clue is mainly in the word 'preview' and partly in the word 'community'.

So what is new? Well, looking at the help file...

Hopping windows. This is an additional temporal window model which was conspicuous by its absence in CTP2. It is, very simply, a contiguous 'batched' window and will be very familiar to users of other CEP products. At any one time the window provides access to the set of events that are alive within that time frame. After a fixed interval the Window is incremented in time to provide access to events in the next fixed period. Because events in StreamInsight are interval-based, an event may appear in more than one time frame. Note that the graphic in the help file appears to be slightly incorrect. Event e1 is unaccountably extended in the first two 'hops'.

Update: After watching Roman Schindlauer's PDC '09 session and re-reading the help file with a little more care, I realised I had not fully understood this new model. This window model is defined by two values. The window has a fixed length, 'S', but also a hop size, 'H'. At each hop, a new window is created of size S.

If H == S (and the two intervals start and end at the same time), you get a special case of a hopping window which Microsoft calls a 'tumbling' window - i.e., a series of contiguous, non-overlapped windows as I described above. If H < S, each window can overlap other windows.

So, my apologies. I didn't properly understand this new model to begin with. However, this doesn't change the observation that the help file graphic needs correcting. Also, the current help file does not provide a comprehensive explanation. The exact behaviour of 'clipping' isn't defined (what about events that appear in multiple windows?), and there is no mention of what happens if you set H > S (which may, of course, not be allowed).

Joins. Again, highly conspicuous by its absence in CTP 2, CTP 3 now supports joins in LINQ! The fact that we have only just got this fundamental capability (what could be more central to CEP than the ability to specify joins directly in your query language) shows just how early a view Microsoft gave us in CTP 2 (CTP 1, incidentally, was a private preview released earlier this year to company employees only).

User-Defined Aggregates (UDAs). This feature allows developers to create highly customised aggregation operators for aggregating over windows. Developers create UDAs by deriving from one of two classes depending on the need to access timestamp information and also following a reasonably straightforward pattern involving creation of an annotated extension method that allows the UDA to be accessed directly within LINQ.

User-Defined Operators (UDOs). UDAs map multiple events to a single scalar value. Like UDAs, UDOs take multiple events as input but can return multiple events. They follow a very similar design pattern to UDAs, but return an IEnumerable<T> rather than a single value.

UDAs and UDOs provide a major piece of the CEP jigsaw and greatly extend the power and expressivity offered by LINQ.

Additional Query Improvements. TopK queries now allow lambdas to be used in Take() to project values from ranked events into the payload of each of the selected events. The Group and Apply operators have also been improved.

In CTP 2, .NET methods in expressions included in LINQ were evaluated at query translation time, which wasn't much use. In CTP 3 they are now evaluated at run time.

CTI improvements. In CTP 2, CTIs could only be enqueued in code in an input adapter. CTP 3 provides support for declarative CTI generation by allowing query template bindings to be configured with an 'AdvanceTime' specifier. You can also configure an adapter factory. CTIs are then generated internally by the engine. CTIs are very cool. Declarative CTIs are very, very cool :-)

Management Service API Improvements. A number of improvements have been made to the management service API to support a richer set of views at different levels within the engine.

So, in summary, Microsoft has made significant progress since CTP2, and CTP 3, as we might hope, looks more like a complete engine. Major gaps in functionality have been filled and we can begin to see more clearly what capabilities will be included in the first release. Keep up the good work

posted @ Tuesday, November 17, 2009 4:12 PM | Feedback (0)

Friday, November 13, 2009 #

C or Java/.NET. Which is fastest? Does anyone know?

The vexed question of language performance came up a couple of times at the ORF conference last month. I've written up my viewpoint on this at http://geekswithblogs.net/cyoung/archive/2009/11/13/136295.aspx. All flames welcome :-)

posted @ Friday, November 13, 2009 5:57 PM | Feedback (0)

Wednesday, November 04, 2009 #

Comparing Event Stream Processing and Rete

Alexandre Alves wrote an interesting article on CEP extensions for Logic Programming (see http://adcalves.wordpress.com/2009/10/30/logic-programming-lp-extensions-for-cep/) in which he mentions the use of Rete-based rules engines. I gave a presentation at a conference last week (October Rules Fest 2009) on the differences and similarities between various event stream processing (ESP) approaches and the Rete algorithm. I had a slide that is not very different in intent to Alexandre's post. I also touched on performance issues, which is something he talks about briefly. I don't know of any solid, well thought out, comparative benchmarking between any of the major ESP and CEP-enabled Rete engines, so I can only hazard informed guesses with regard to likely performance issues. I thought it would be useful to summarise some of my thinking in this area.

posted @ Wednesday, November 04, 2009 6:49 PM | Feedback (0)

Friday, October 30, 2009 #

October Rules Fest 2009: Day 5

My report on day five (the last day) of the October Rules Fest 2009 is at:

http://geekswithblogs.net/cyoung/archive/2009/10/30/135901.aspx

posted @ Friday, October 30, 2009 11:59 AM | Feedback (0)

October Rules Fest 2009: Day 4

My report on day four of the October Rules Fest 2009 is at:

http://geekswithblogs.net/cyoung/archive/2009/10/30/135855.aspx

posted @ Friday, October 30, 2009 5:32 AM | Feedback (0)

Wednesday, October 28, 2009 #

October Rules Fest 2009: Day 3

My report on day three of the October Rules Fest 2009 is at:

http://geekswithblogs.net/cyoung/archive/2009/10/28/135780.aspx

posted @ Wednesday, October 28, 2009 6:07 PM | Feedback (0)

Tuesday, October 27, 2009 #

October Rules Fest 2009: Day 2

I’m at day two of the October Rules Conference at the Adolphus Hotel in downtown Dallas. My report on the day's processings is at:

http://geekswithblogs.net/cyoung/archive/2009/10/27/135756.aspx

posted @ Tuesday, October 27, 2009 6:20 PM | Feedback (0)

Friday, October 09, 2009 #

BizTalk Server 2006 R2 - Service Pack 1 announced

Microsoft today announced Service Pack 1 for BizTalk Server 2006 R2. Note that this is for the previous version of BizTalk Server, not the current version (BizTalk Server 2009). No doubt a service pack for BTS 2009 will follow in due course.

The service pack is currently at beta. It has only been announced, not released. You can download the beta if you log onto the Connect site.

See http://blogs.msdn.com/biztalkcrt/archive/2009/10/09/announcing-biztalk-2006-r2-sp1.aspx

There are a couple of new features. See http://msdn.microsoft.com/en-us/library/ee532481(BTS.20).aspx.

For a list of bug fixes, see http://support.microsoft.com/kb/974563.

posted @ Friday, October 09, 2009 1:36 PM | Feedback (1)

Wednesday, October 07, 2009 #

October Rules Fest 2009. See you in Dallas

I've largely finished my presentation for the October Rules Fest 2009 conference in Dallas at the end of the month. I'm speaking on complex event processing (CEP). My plan is to provide a broad survey of CEP technologies, chiefly concentrating on the similarities and differences between event stream and rules processing. There has been a lot of interest and activity around event processing in the rules community in recent years, and not a little controversy about the best approaches and, indeed, the role, if any, of Rete rules engines in detection of complex events. Constructing the presentation has been something of a journey for me, and hopefully it will prove of interest to those attending the conference.

This is rather last-minute plug for ORF 2009. It is the second year of this 'alternative' rules conference which is differentiated by a clear and unashamed focus on technology (there are other well-established rules conferences which focus on application at the business level), and by a wide-ranging interest in several related areas of science and research. For me, some of last year's highlights included hearing Gary Riley explain how he managed to squeeze so much performance out of the CLIPS engine and listening to Dan Levine's talk on rule-based mechanisms in the human brain. It's that kind of event.

The major rule-processing vendors (ILog (now IBM), FICO, Tibco, etc.,) are well represented at the event together with the JBoss team. Charles Forgy, who came up with the Rete algorithm three decades ago, is a star speaker (a fascinating talk is promised on how to maximise the benefits of parallelisation in rules engines). I'm particularly looking forward to hearing Andrew Waterman's talk on the use of rules processing in game-playing software used to promote sustainability and development of natural resources in Mexico. I've been aware of this project for some time. Greg Barton will be reporting on his experiences at Southwest Airlines. There are interesting sessions on rule modelling and aspects of rule languages and DSLs, plenty on CEP, and various talks on constraint programming, rule verification and other topics. And, to remind us all that technology, for technology's sake, is never a good idea, John Zackman will be there to talk about the role of rules in Enterprise Architecture.

ORF 2009, only in its second year, offers an incredibly varied diet for the rule technologist. Together with the boot camps and introductory sessions at the beginning of the programme, it offers practical hands-on experience, a chance to learn about rules processing in depth, a showcase for the wide-ranging application of rules in many different areas of IT and an insight into many areas of research.

Places are still available, I understand. The cost is kept as low as possible by the conference organisers, so visit http://www.octoberrulesfest.org for more information and book in while you can.

posted @ Wednesday, October 07, 2009 10:19 AM | Feedback (15)

Monday, September 28, 2009 #

Serialising Microsoft StreamInsight QueryTemplates

At first sight, StreamInsight doesn’t appear to offer any support for dynamic queries. Once you have got over the initial ‘wow’ factor of creating continuous queries over streams using LINQ, rather than some SQL dialect or specialised DSL, the reality hits you that your query is just code. You wrote the LINQ using C# or VB.NET, and you ended up compiling it into some assembly.

That’s all very well, but it’s not very flexible. How are you going to store and manage your queries in a repository so that you can review, change and version them? Does Microsoft really expect you to recompile and re-deploy assemblies every time you want to create or change a query in StreamInsight? Surely you should be able to dynamically deploy queries straight from a repository.

Have no fear. All is well. The QueryTemplate is serialisable. It isn’t serialisable in the .NET sense (i.e., it isn’t marked up as serilaisable, and it doesn’t implement ISerializable). Rather, it implements the QueryTemplate.Definition property which returns an XML document containing a complete rendering of your template, complete with a full dataflow and all the operators. To deserialise the XML, you use Application.CreateQueryTemplate. You have to pass in an XmlReader that you have created over the XML. You get back a clone of the original QueryTemplate which you can now bind to input and output adapters and emit to the engine as a Query. Nothing could be simpler.

There are a couple of gotchas. First, don’t do what I did while experimenting. I created a QueryTemplate and serialised it to XML. I then tried to deserialise it as a second, identical, QueryTemplate in the same Server instance. I got back an InvalidDefinition error. It seems you can’t have identical QueryTemplates in a single StreamInsight Server instance.

The second thing to note is that the serialised QueryTemplate does not contain fully qualified references to your event types. You still have to define the event types in your application by calling the CreateEventType method. If you plan to store QueryTemplate XML in some custom repository, you will also need to store the fully qualified names of the .NET types your have defined for your events. Of course, you will also need to ensure that these types are available at runtime.

Much the same argument applies to adapters. The QueryTemplate XML does not contain any information on which input and output adapters you want to bind to your query. However, once you realise that your QueryTemplates are serialisable, it is simple to work out how you might store and manage the template in some repository together with event type and adapter information. All this information, taken together, defines the StreamInsight concept of a query. You can write code to extract it from the repository at run-time and run the query.

The current CTP is just a glorified SDK. Will Microsoft provide a repository and tooling in the release version? I have no idea. Given the freedom StreamInsight offers in terms of event definition, and given the fact that the engine can be hosted in custom applications, it is difficult to see how Microsoft could provide a totally generic end-to-end solution that would support externalisation of queries in any and all circumstances. They could provide a repository and API, and leave it to developers to write the code to exploit the repository. However, you would still need to write the LINQ together with code to create QueryTemplates and extract and upload XML to the repository.

posted @ Monday, September 28, 2009 7:26 PM | Feedback (6)

Wednesday, September 23, 2009 #

A proposed introduction...

What a pity. I’ve spent some time recently evaluating SQL Server 2008 Reporting Services with respect to its proposed use on a UK public sector project that will implement a part of the 'national infrastructure'. In every respect but one, Reporting Services provides an excellent fit for the stated requirements. There is already a commitment to using SQL Server, and SharePoint, within the project, so Reporting Services appears the way to go.

What is the exception? Well, Reporting Services provides a nice little tool called 'Report Builder' for creation, amongst other things, of ad hoc reports. This is a user-friendly little gizmo that can be downloaded directly from the portal via Microsoft's excellent ClickOnce technology. ClickOnce ensures that the application is installed in a sandboxed fashion under the current user's profile, and does not interfere with other installations for other users.

All good stuff, except that the proposed system needs to provide ad hoc reporting facilities to a number of different agencies across the UK. Oh, and it must be formally accredited by the security people before it can go live. It turns out that there is no prospect that they will agree to installation of any software on client desktops at those agencies. Not even using ClickOnce. It’s out of the question.

SAP offers a similar toolset to Reporting Services, but they have a tool for designing ad hoc reports that is entirely browser-based and has zero footprint on the client.

Mmmm. Time for the Microsoft SQL Server product team to meet Ajax, methinks.

posted @ Wednesday, September 23, 2009 11:47 AM | Feedback (4)

Charles Young

News

Twitter

Article Categories

Archives

Post Categories

Image Galleries

Alternative Feeds

BizTalk Bloggers

BizTalk Sites

CEP Bloggers

CMS Bloggers

Fun

Other Bloggers

Rules Bloggers

SharePoint Bloggers

Utilities

WF Bloggers