A question came up tonight on BizTalkGurus on my favourite subject of rule engines. I don’t blog enough these days, so this gives me an excuse. Essentially, the question concerned an incorrect, but understandable, suspicion that MS BRE may be using remoting to execute rule sets out-of-process. This is not the case. I hope the author of the original request won’t mind me repeating his question in a more open forum.
“When using the Policy class, the Rule Engine instance is “fetched” from inside a cache which is stored in memory and maintained by the Rule Engine Updater Service. The REU Svc, uses .Net Remoting to pass a “REFERENCE” (MarshallByRef) proxy class of the RuleEngine Instance to the Policy object. The Policy then invokes the Rule Engine instance through this “Reference Proxy”. Well, in this week’s consulting project, my students at Xterprise reminded me that inside of .Net Remoting, when you pass an object by reference, it’s not the actual object. It’s a reference and the calls are marshaled to the live object over Http or Tcp channel. The channel in this case is TCP. I remember this. Thus according to the rules of .Net remoting, this would definitely mean that the actual execution of the Rules are on the server where the REU Svc resides, in the case of the Policy object.
Now for the PolicyTester class, this is different. The PolicyTester does not obtain a RuleEngine Instance from the Cache. It just creates the RuleEngine instance in the host that creates the PolicyTester object. The RuleEngine instance in turn creates the “Network”/Rete classes and executes them within the host that created the object. This is why the Business Rules Composer can “Test” policies without them being Published and Deployed, and without the REU Svc started. It uses the PolicyTester class which runs the rules in the hosts’ process.
Can someone verify my thinking and research here… Any pointers, corrections would greatly be appreciated.”
I've always maintained that that the PolicyTester class is poorly named. If MS BRE was licensed in a more reasonable manner (i.e., not linked inextricably by legality to installations of BizTalk Server), this class would play a far more central role as the means for obtaining and invoking rule sets. Instead, its importance is downplayed. Much, much worse than that, Microsoft provides no support for it in the integration between MS BRE and BizTalk Server. Again, they don't support it in the integration they provide with RFID Server (which is really unforgivable given the need for pattern-matching rule execution over large data sets in many RFID scenarios). This, combined with generally poor tooling, makes it far harder to develop with the rules engine than it should be. I've come to see the PolicyTester class as the more basic and fundamental of the two classes, and nowadays I generally use my own home-grown code to invoke rules within BizTalk Server so that I can exploit PolicyTester during development.
The central point here is to understand the various representations of rules sets at different stages. There are, broadly speaking, three different representations. Rules are generally created and captured in some external form which I will term form ‘A’. This could be any format you wish. However, using the Rules Composer, rule sets are captured using a proprietary Microsoft XML rule language (MS BRL). The resulting XML documents are generally stored in the SQL Server repository provided by Microsoft. This is a very basic rule repository indeed, and captures rules as complete XML text documents. This simplicity, in turn, makes it virtually impossible to build a reasonable rule development tool over the repository, and partly explains why the Rule Composer is so hard to use in development. You should understand, though, that the repository and Rule Composer are simplistic tools that exploit the underlying Rules Framework supplied by Microsoft. You could integrate other repositories and tools into the same framework. In fact, in a little known feature of the MS Rules Framework, you are actually provided with pre-built functionality for extracting rule sets from other repositories using OLE-DB. I’ve personally yet to see anyone exploit this.
It turns out MS BRL is really the serialised form of rule sets expressed using a rule set document object model (DOM). Microsoft has implemented custom de-serialisation code for this purpose in the BusinessRulesLanguageConverter class. They don't support direct XML serialisation using XmlSerializer although you can use a formatter to serialize rule sets. However, this won't result in serialisation to MS BRL. Neverthless, there is a direct association between MS BRL and Microsoft's Rule DOM. Rule DOM objects are, then, the second form of rule set representation. I will term this form ‘B’. The Rule DOM objects exist in memory. That is all very well, but these in-memory rule sets are just documents - they are not directly executable. They need to be passed through a translation stage in order to create something which is executable. The MS Rules framework decouples these different concerns. For example, a rule set in form 'B' is configured to select a specific translator which, by default, is Microsoft’s RuleSetToReteTranslator class. Have a very careful look at the property list in the Rule Composer, and you will discover that, because you are using MS BRL as the form ‘A’ representation, and because MS BRL de-serialises into Microsoft's Rule DOM, you are allowed to configure a specific translator there if you wish. You would have to build your own translator, of course, and the main reason for doing this would be to target a different rules engine. I did experiment with this a long time ago, and proved that the feature is functional. Again, I’ve yet to see it used in real-life. In passing, I should state that I am actually profoundly sceptical with regard to the practicability of portability of rule sets between different engines, but that is another matter.
A translator is responsible for the third representation of a rule set which I will term form ‘C’. This is the representation of a rule set in some executable form. For MS BRE, this is an in-memory Rete (a node network) composed of inner alpha and beta networks containing Class, Select, Join and Terminal nodes. This representation (a fairly text-book implementation of Rete) is specific to MS BRE and has nothing at all to do with the Rule Engine Update Service. I got confused about this myself when I first started looking at the internals, but trust me. The REU service is deliberately and carefully decoupled from any specific engine. This is a central architectural aspect of the design of the Rules Framework, although no-one really exploits it.
The Rules Engine Update (REU) Service works with pub-sub adapter components in order to subscribe to changes to rule sets in form 'A'. Instances of this Windows service run on individual servers (again, due to licensing, rather than technical, restrictions, these have to be BizTalk servers) The pub-sub adapter component for the SQL Server repository is actually not implemented as true pub-sub (the SQL Server repository does not publish change notifications). Instead, it polls at regular intervals configurable in the registry. This is a real pain when developing because each time you publish and deploy a new version of a rule set, you have to wait a minute or so before the change notification is pulled down to the REU Service.
When the REU Service obtains a deployment notification (e.g., for a new version of a rule set), it passes this on to registered engine instances. When you use the Policy class to invoke a rule set, the Policy obtains an instance of a rule engine from a local engine cache. The rule engine, in turn, works with a local rule set cache. Because the Policy class does not provide a rule set, the engine uses the REU service to retrieve the latest version of the rule set and registers for deployment notifications from the REU service. Each time the REU Service receives a notification, it calls back to the engine so that the local caches can be cleared. The engine asks the REU Service for the new version of the rule set. The REU Service, incidentally, maintains its own local cache of form 'A' rule sets.
The REU Service uses a RuleStore component to get up-to-date versions of rule sets. A RuleStore is implemented to understand a specific external rule store. Typically, you will be using the Microsoft-supplied SqlRuleStore. This component fetches the rule set in form 'A' and converts it into form 'B'. The REU Service caches the rule set locally in form 'A'. Each time a rule engine is instantiated either by the Policy object or after the deployment of a new version of a rule set, it obtains the form 'B' rule set from the REU Service and uses the configured translator (in this case, the RuleSetToReteTranslator) to convert the rule set into an executable form (form 'C'). The translator component returns an executor object which is used by the rule engine instance to execute the rule set.
So, we have different layers of caching. Rule engine instances hold executors which, in turn, reference executable Retia (form 'C'). Engine instances are cached locally along with rule sets in form 'B' as Rule DOM object graphs. This allows MS BRE to avoid using the REU Service every time it spins up a new engine instance (remember that in an environment like BizTalk Server, you may have lots of engine instances running simultaneously for the same rule set - these instances are single-threaded and must isolate their data from other engine instances). The REU Service itself knows nothing about the runtime executable form 'C' rule sets and, in answer to the original question, there is absolutely no remoting going on at the execution stage. The REU Service caches the form 'A' rule sets, and refreshes the cache when notified of a change in the external store (it also regularly purges its cache).
I hope that all makes sense. When you use the PolicyTester class, things are much simpler. PolicyTester is used to instantiate rule engine instances directly over form 'B' rule sets without going to the REU Service. You have to write some additional code to instantiate the form 'B' Rule DOM rule set yourself. Typically, you use the SqlRuleStore class to get a form 'A' rule set for the SQL repository and convert it into form 'B'. In more advanced scenarios, you might construct rule sets dynamically using Microsoft's Rule DOM. It is trivial to write your own RuleStores. For example, for experimentation and implementing standalone demos, I often use a very simple little RuleStore I created that obtains form 'A' rule sets from managed resources. I then de-serialise these into form 'B'.
Here is a diagram that attempts to put everything together:
I have simplified the explanation somewhat, as the Rules Framework offers a number of additional classes and components that are used to handle the different representations of rule sets. For example, I’ve not described the use of RuleSetDeploymentDriver components or custom RuleLangugeConverters. The MS Rules Framework was a spirited attempt by MS to create a wide-ranging environment that could integrate rules held in different forms in different repositories, mange the deployment of rule sets out across an enterprise environment and even target a range of different rules engines. It’s sad to see that the potential power of this framework has never really been exploited and that it has not been developed further over the years.