Being known for my interest in rules processing, I quite often get asked to help with problems with MS BRE. A couple of days ago, I was asked to help investigate an issue occurring in production for a BizTalk Server application. Occasionally, in a fairly high throughput system, BizTalk logs an error stating that a problem has been encountered while executing a rule set. That is the only information provided, with no hint of what the problem might be, and because the issue only occurs intermittently under real-world conditions in the production environment, it was not obvious how to obtain further insight without disrupting live operations.
The MS BRE is designed to be hosted within .NET applications as an in-process library. Its approach to handling error conditions during rule set execution is to throw all exceptions back to the application that invoked execution. The first question I was asked was what registry setting could be used to turn on event log tracing for the engine. I had to report that no trace functionality exists at all within the engine, although the Rule Engine Update service does log exceptions. Although emitting exceptions back to the host application makes sense, MS BRE really ought to provide the ability to switch additional event log tracing on for diagnostic purposes. If the host application does not log the details, exceptions become opaque.
We were getting no detailed information on the error because the application-level exception handling code does not recurse through inner exceptions in order to log as much information as possible. The Call Rules shape in BizTalk orchestrations uses the Policy class to invoke a rule set. When a Policy object catches an exception, it wraps it in an outer PolicyExecutionException which it then throws. The information we needed was assigned to the inner exception.
The best approach would be to bite the bullet and deploy a new version of the BizTalk application with improved error logging. If we are lucky, this may only require updating some centralised error logging assembly. However, there is a good chance that we may need to deploy a whole new version of an orchestration. Either way, there will probably be some disruption to the live service because we will probably end up stopping and starting the BizTalk application in order to load new assembly versions. I suppose, thinking about it, we could try using side-by-side assembly deployment as an alternative strategy, but I’m not keen on that idea.
The problem I was posed was to find a way of getting richer error information from the rules engine with minimum disruption to the live system. As we have established, there is no nice feature in the engine that I can simply switch on with a registry setting. However, all is not lost. In this article, I will describe another approach that allows you to control and configure the logging of rule engine exceptions using the straight-forward, but little known, 'compensation' feature of the rule engine. As we will see, this feature can be used this to introduce error logging without stopping the live system.
Given that the issue here would be best resolved by introducing better error logging at the orchestration level, you may well question the relevance of this article. However, it will serve a number of purposes:
a) To demonstrate one way of solving this particular dilemma when supporting existing production environments. That may possibly be of help to someone out there.
b) To demonstrate the general mechanics of the rule engine compensation feature.
c) To build an understanding of where the compensation feature might be used for more advanced purposes.
Along the way, I will also use this as an opportunity to illustrate some additional information concerning a previous article on the various ways in which rule sets are represented in the repository, the Rule Engine Update (REU) service and the engine itself. See:
Rule Engine Compensation
In regard to the rules engine, the term 'compensation' is, frankly, a little misleading. In BizTalk orchestrations, WF workflows and the BPEL specification, compensation refers to built-in support for a set of exception handling patterns based broadly on the 'Saga' model and associated with the concept of transactions. There is no equivalent notion of transactions within the rules engine, and the compensation mechanism is really just a call-back facility that allows rule engine exception handlers to notify custom code of exceptions before re-raising those exceptions. As it happens, the relevant event handlers in the MS BRE are associated with discrete engine functions such as assertion and deletion. However, I cannot fully defend the term 'compensation' in this context, especially as the engine does not pass information about the current engine operation when it performs a callback.
As you might expect, the callback mechanism is implemented through the use of delegates. An application passes a delegate (which may, of course, be a multicast delegate) to the engine in order be notified of exceptions. The delegate is of type RuleEngineCompensationHandler, and takes two arguments. The first is the exception object and the second is a 'user data' object. This can be of any type, and is registered with the rule engine along with the delegate. It is passed back to the delegate by the engine when an exception occurs. The user data object allows a host application to pass arbitrary data to compensation handlers.
Here is an example of a very simplistic logging method that is compatible with the RuleEngineCompensationHandler delegate in which a custom user data object is used to pass an event ID. The code uses the System.Diagnostics.EventLog class:
public bool LogException(Exception ex, object userData)
{
// Initialise data
bool retVal = false;
string sourceName = "MSBusinessRulesEngine";
string message = "An exception occurred when executing a " +
"rule set.\r\n" + ex.ToString();
EventLogEntryType entryType = EventLogEntryType.Error;
// Assume the userData object is a CustomUserData object.
int eventId = ((CustomUserData)userData).EventId;
// Create the source, if it does not already exist.
if (!EventLog.SourceExists(sourceName))
{
// Create an EventLog instance and assign its source.
EventLog newLog = new EventLog();
newLog.Source = sourceName;
}
try
{
EventLog.WriteEntry(sourceName,
string.Format(CultureInfo.CurrentCulture, "{0}: {1}",
DateTime.Now.ToString(), message),
entryType, eventId);
}
catch (Exception)
{
throw;
}
return retVal;
}
Delegates are associated with user data objects. This association is represented by the use of CompensationHandlerInfo objects. These are created by the host application initialised with a delegate and a user data object and then passed to the rule engine in order to hook in the compensation handler code.
After the Call Rules shape in BizTalk orchestrations, the second most common way of invoking the rules engine is via the Policy class. Indeed, as I noted above, the Call Rules shapes uses the Policy class internally. A Policy object represents a version of a rule set associated with a rule engine instance, and is used to execute the rule set against a set of facts. The Policy object, however, does not support the registration of compensation handlers, so to use this feature you must exploit lower-level objects. Microsoft's engine operates in the context of the Microsoft Business Rule Framework which defines a number of classes and interfaces representing various abstractions. These include the RuleEngine class and executor objects that implement the IRuleSetExecutor interface. It is generally assumed that the RuleEngine class explicitly represents Microsoft's engine, but this not the case. It actually represents the abstract concept of a rule engine, but does not implement any rule processing functionality itself. The RuleEngine class is defined as part of the framework, and the framework is not tied to any one specific rule engine technology.
A RuleEngine instance uses an executor object to execute a rule set. Executor classes implement the IRuleSetExecutor interface, and provide access to the specific functionality of a given engine. The RuleEngine class is a text-book example of the adapter design pattern. It wraps the executor, exposing methods that largely correspond to the IRuleSetExecutor interface. However, it does not implement this interface itself.
Compensation handlers can be registered using either a RuleEngine object or an executor. However, the MS BRE Rete executor is an internal class, and is not directly accessible to custom code. When writing custom code to configure compensation, you will therefore use a RuleEngine object. Later in this article we will see that executors can, in fact, be passed to custom code indirectly from within a rule.
Registering compensation from within external custom code doesn’t really make too much sense. Because all exceptions are thorwn back to your code, you might just as well trap any exceptions and handle them at that level. This is probably why the Policy class does not support compensation handler registration. In a addition, when writing code to use the RuleEngine object, you will often want to obtain rule sets via the Rule Engine Update (REU) service in the same way the Policy class does. This is entirely achievable, but with issues.
Here is some simple code that demonstrates how to obtain the latest version of a rule set from the REU service and to configure a rule engine instance to use a compensation handler. The code uses the LogException handler defined above. Given my observation about the limited usefulness of this approach, you might wonder, why I have bothered to provide a code example. One reason (excuse?) is that it helps illustrate a previous article I wrote on the various forms in which a rule set is expressed as it moved from a repository to the engine via the REU Service. The code demonstrates how to obtain rule sets directly from the REU service and how the framework supports conversion of MS BRL into RuleSet object graphs using the IRuleLanguageConverter interface implemented, in this case, on the BusinessRulesLanguageConverter object.
...
using System;
using System.IO;
using System.Xml;
using Microsoft.RuleEngine;
using Microsoft.RuleEngine.RemoteUpdateService;
...
public ExecuteRuleSet()
{
RuleSet rs = null;
// Create a proxy for the REU Service
RemoteUpdateServiceProxy proxy
= new RemoteUpdateServiceProxy();
// Get latest version of rule set
RuleSetInfo rsi = proxy.GetLatest("RuleEngineLoggerExample");
// Create a language converter for MS BRL
BusinessRulesLanguageConverter msbrlConverter
= new BusinessRulesLanguageConverter();
try
{
// Check that we have permission to the rule set
if (proxy.IsRuleSetAccessible(rsi))
{
// Get the raw MS BRL
byte[] msBrl = proxy.GetDefinition(rsi.Name,
rsi.MajorRevision,
rsi.MinorRevision);
// Convert the MS BRL into a ruleset object graph
// NB: There is a close association between MS BRL
// and Microsoft’s Rule DOM, so this really
// constitutes a form of fairly direct de-serialisation
// using the converter.
using (MemoryStream msBrlStream
= new MemoryStream(msBrl))
{
RuleSetDictionary ruleSets;
VocabularyDictionary vocabs;
msbrlConverter.Load(msBrlStream,
out vocabs, out ruleSets);
// Extract the required rule set
rs = ruleSets[rsi.Name];
}
}
}
catch (Exception ex)
{
throw new ApplicationException(
"Could not load rule set.\r\n" + ex.Message, ex);
}
// Check we were successful
if (rs != null)
{
// Create a RuleEngine instance.
// RuleEngine represents the abstract concept of
// a rule engine. The rule set passed to the constructor
// has configuration data which, by default, configures
// the rule set to be translated using the
// RuleSetToReteTranslator. This component creates
// an MS BRE Rete object which is the executor. An
// executor accesses a concrete implementation of a
// rule engine.
RuleEngine re = new RuleEngine(rs, true);
// Create a custom user data object to hand information
// to the RuleEngine_LogException handler
CustomUserData custUserData = new CustomUserData();
custUserData.EventId = 2500;
// Create a CompensationHandlerInfo to bind a delegate
// to the user data
re.CompensationHandlerInfo
= new CompensationHandlerInfo(LogException,
custUserData);
// Create the facts
object[] facts = new object[1];
XmlDocument xDoc = new XmlDocument();
xDoc.Load(@".\ExampleFiles\po.xml");
// NB. Ensure the document type is correct here. This
// can be obtained by inspecting the root element in the
// schema using the Rules Composer.
facts[0] = new TypedXmlDocument("po.PurchaseOrder", xDoc);
try
{
// Execute the rule set without tracking
re.Execute(facts);
}
catch (Exception ex)
{
// It would generally be better to do exception
// logging here rather than use compensation
throw new ApplicationException (
"The application has received an exception " +
"from the rules engine,\r\nbut can’t be " +
"bothered to provide you with any details!!");
}
// Save Typed XML
xDoc.Save(@".\ExampleFiles\po.xml");
}
}
This code suffers from a specific problem. When you use the Policy class to invoke a rule set, you get the benefit of caching. There are two caches involved. The first stores RuleSet objects in order to avoid repeated calls into the REU service. The second caches RuleEngine instances in order to speed up second and subsequent invocations of a rule set. Unfortunately, the two caches are internal, and not accessible to custom code. Hence, the example code cannot access the caches and is really quite inefficient in comparison to using the Policy object. There are a couple of ways you might remedy this. The first would be to use reflection to access the internal caches. The second would be to extend the above code to support custom caching.
As you can see, using the RuleEngine object in custom code poses some challenges. It would be better to write code that uses the Policy object and use local exception handling to log errors. However, this would be too disruptive for our purposes because it means changes to the orchestration code and re-deployment of those orchestrations to the live environment. Unless side-by-side deployment techniques are used, this would mean taking the production environment down after first ensuring that any long-lived processes are completed. Fortunately, there are better ways that require significantly less disruption.
Using a Fact Retriever
We want to introduce logging in a new version of a policy without the need to redeploy an updated version of the live BizTalk application. There are two ways in which we can achieve this. The first is to use a Fact Retriever component. Fact Retrievers are a really useful feature of MS BRE. They are helper components that can be used to assert facts to the engine each time a rule set is executed. They can be used alongside the more common approaches to asserting facts (e.g., passing an array of facts to the Execute method of the Policy object or directly using the Assert method of the RuleEngine object). Their power derives in part from the ability to configure a Fact Retriever declaratively as part of a rule set without writing any additional code (except, of course, the code that implements the Fact Retriever itself). In addition, Fact Retrievers support a simple handle value. When a RuleEngine object is first instantiated and executed, it calls the UpdateFacts method of the configured Fact Retriever and provides a null handle. Importantly for our purposes, it also passes a reference to the RuleEngine object. In the typical design pattern, the Fact Retriever assigns some custom object as the handle.
When execution of the rule set completes, the engine automatically retracts all remaining facts that have been directly asserted, but retains any facts that have been asserted by the Fact Retriever. This is why these facts are called ‘long-term’ facts. The RuleEngine instance is cached for future re-use. When, at some future time, the RuleEngine instance is retrieved from the cache and re-executed, the long-term facts are still within the working memory. The engine re-invokes the UpdateFacts method of the Fact Retriever, passing back the handle that was assigned in the first invocation. The handle, therefore serves two broad functions. The first is to allow the fact retriever to track executions of the rule set. Typically, this is used to differentiate between the first and subsequent executions. The second function is to round-trip data between executions in order to control fact retrieval. There are all kinds of ways this can be used. For example, a Fact Retriever might decide to re-assert long term facts based on the length of time that has elapsed since the last invocation by recording and updating a timestamp on the handle object.
Each time the engine invokes the UpdateFacts method, the Fact Retriever can decide what facts it wishes to assert, retract or update. However, it can also access other RuleEngine methods and properties, including the CompensationHandlerInfo property we used in the example code earlier. This means that we can initialise a rule engine instance with a compensation handler. A simple example is shown below:
using System;
using System.Globalization;
using System.Diagnostics;
using Microsoft.RuleEngine;
class ExceptionLoggerInitialiser : IFactRetriever
{
public object UpdateFacts(RuleSetInfo rulesetInfo,
Microsoft.RuleEngine.RuleEngine engine,
object factsHandleIn)
{
object factsHandleOut;
if (factsHandleIn == null)
{
// Create a custom user data object to hand information
// to the RuleEngine_LogException handler
CustomUserData custUserData = new CustomUserData();
custUserData.EventId = 2500;
// Create a CompensationHandlerInfo to bind a delegate
// to the user data
engine.CompensationHandlerInfo
= new CompensationHandlerInfo(LogException,
custUserData);
factsHandleOut = custUserData;
}
else
{
factsHandleOut = factsHandleIn;
}
return factsHandleOut;
}
private bool LogException(Exception ex, object userData)
{
// Initialise data
bool retVal = false;
string sourceName = "MSBusinessRulesEngine";
string message = "An exception occurred when executing a " +
"rule set.\r\n" + ex.ToString();
EventLogEntryType entryType = EventLogEntryType.Error;
int eventId = ((CustomUserData)userData).EventId;
// Create the source, if it does not already exist.
if (!EventLog.SourceExists(sourceName))
{
// Create an EventLog instance and assign its source.
EventLog newLog = new EventLog();
newLog.Source = sourceName;
}
try
{
EventLog.WriteEntry(sourceName,
string.Format(CultureInfo.CurrentCulture, "{0}: {1}",
DateTime.Now.ToString(), message),
entryType, eventId);
}
catch (Exception)
{
throw;
}
return retVal;
}
}
public class CustomUserData
{
private int _eventId;
public int EventId
{
get
{
return _eventId;
}
set
{
_eventId = value;
}
}
}
All that is now necessary is to compile the assembly with a strong name, install it into the GAC on the server and configure a new version of the rule set to use the Fact Retriever. Deploy the new version of the rule set and wait for exceptions to be logged!
Using a Rule to configure the Executor
The Fact Retriever approach outline above is, in my opinion, the best way to solve the problem in hand. However, there is a second approach. It’s worth describing this because it leads nicely into understanding more advanced uses of compensation handlers in rule sets. The second approach is a less preferable than the Fact retriever approach because it requires changes to rules in existing rule sets, and introduces a correspondingly greater risk of breaking the application.
We saw earlier that the classes that implement the MS BRE engine are marked as ‘internal’. Specifically, this includes the RuleSetToReteTranslator class (an implementation of IRuleSetTranslator) and the Rete class (an implementation of IRuleSetExecutor) produced by RuleSetToReteTranslator. The interfaces implemented by these two classes are part of the rules framework, and can potentially be used to enable the integration of other engines into the framework. They are, therefore, marked as ‘public’. This means that, although we cannot directly instantiate or manipulate the MS BRE Rete class from custom code, we can certainly access a Rete object via its executor interface, assuming we can somehow obtain a reference to that interface.
Microsoft’s engine implements a number of pre-defined ‘engine operations’. These can be used, as required, within rule conditions and actions, and will be very familiar to anyone who has written MS BRE rules. The easiest way to explore them is via the Rules Composer. Microsoft has defined two vocabularies, one listing friendly names for engine ‘functions’ and the other listing names for engine ‘predicates’ (methods that return a Boolean value). Engine functions include the familiar ‘Assert’, ‘Update’ and ‘Retract’ functions as well as a small library of additional commonly-used functionality. One of the more mysterious engine functions is called ‘Executor’. It returns an instance of the current executor typed as IRuleSetExecutor. Because you will almost certainly be using Microsoft’s built-in engine functionality, this will really be an instance of the internal Rete class.
You typically use the Executor engine function to pass an instance of the executor out to custom code. You can create a method in custom code, and define it with an IRuleSetExecutor parameter. At run-time, you will assert an instance of the method’s class to the engine, even if your method is static. This, of course, assumes you have not set the StaticSupport registry key to a value greater than 0. I strongly recommend that you do not use the StaticSupport feature of MS BRE unless there is a compelling reason to do so. It introduces a high risk of breaking rule sets that were created under the default setting. The resulting run-time failures are very opaque and difficult to diagnose.
You can now invoke your method within your rules. This may be done using a rule action, or alternatively a predicate or argument in a rule condition. You use the Executor engine function to assign a value to the method’s parameter. In your custom code, you can now access the IRuleSetExecutor methods of the executor class, including the SetCompensationHandler method.
We can use the Executor engine function to enable logging. The basic idea is to fire an initial rule that will configure the executor with a compensation handler. To do this, you first need to create some helper classes and methods. The code will be very similar to the equivalent Fact Retriever code. The following provides a simple example.
using System;
using System.Diagnostics;
using System.Globalization;
using Microsoft.RuleEngine;
public class RuleEngineExceptionLogger
{
private bool _isInitialised;
private CompensationHandlerInfo _compHandlerInfo;
public bool IsInitialised
{
get
{
return _isInitialised;
}
}
public void SetCompensationHandler(IRuleSetExecutor executor)
{
if (executor != null)
{
CustomUserDatauserData = new CustomUserData();
userData.EventId = 2500;
RuleEngineCompensationHandler reCompHdlr =
new RuleEngineCompensationHandler(LogException);
_compHandlerInfo =
new CompensationHandlerInfo(reCompHdlr, userData);
executor.SetCompensationHandler(_compHandlerInfo);
_isInitialised = true;
}
}
public bool LogException(Exception ex, object userData)
{
// Initialise data
bool retVal = false;
string sourceName = "MSBusinessRulesEngine"
string message = "An exception occurred when executing a " +
"rule set.\r\n" + ex.toString();
EventLogEntryType entryType = EventLogEntryType.Error;
int eventId = ((CustomUserData)userData.EventId;
// Create the source, if it does not already exist.
if (!EventLog.SourceExists(sourceName))
{
// Create an EventLog instance and assign its source.
EventLog newLog = new EventLog();
newLog.Source = sourceName;
}
try
{
EventLog.WriteEntry(sourceName,
string.Format(CultureInfo.CurrentCulture, "{0}: {1}",
DateTime.Now.ToString(), message),
entryType, eventId);
}
catch (Exception)
{
throw;
}
return retVal;
}
public class CustomUserData
{
private int _eventId;
public int EventId
{
get
{
return _eventId;
}
set
{
_eventId = value;
}
}
}
We will now consider how to structure the rule set. We need to configure the executor using a call to the SetCompensationHandler method of our helper class. We can create a Fact Retriever to assert an instance of the helper class to the engine, and configure the rule set to use that Fact Retriever. We then need to ensure that the SetCompensationHandler method is invoked before the main rule set processing is undertaken.
Custom methods can be invoked as rule actions, or as predicates or predicate arguments in a rule’s conditions. The MS BRE uses an internal match-resolve-act cycle which separates the evaluation of rule conditions from the firing of rule actions. Conditions are evaluated during the match phase, but actions are only executed during the act phase. Hence, if we simply add a rule that invokes the SetCompensationHandler as an action, this will not be sufficient. The engine will have evaluated all relevant rule conditions before logging is configured. If any error is thrown during the initial match phase, it will not be logged.
We could consider changing the SetCompensationHandler to return a Boolean value and then using it as a predicate in a rule condition. However, this won’t work either. You cannot control the order in which rules and conditions are evaluated. In a Rete engine, condition evaluation is conducted without reference to the rules in which those conditions appear. Individual rules only map directly onto the results of the match phase, and not onto the match phase itself. Hence, it makes no sense to talk about evaluating rules in any specific order. Although, for the most part, MS BRE will evaluate conditions broadly in the order they appear in a given rule, this is by no means guaranteed. A condition that appears in multiple rules will be evaluated just once. Also, MS BRE tries to arrange the Rete network in an optimal fashion. There is a little-known ‘hint’ system which rule developers can use to help the engine with this optimisation (I intend to discuss this in a future article). Nothing is certain in regard to the order in which rule conditions are evaluated. Hence, if we use a modified version of the SetCompensationHandler as a predicate method in a rule condition, we have no control over when this method will be invoked in relation to other conditions. Hence, any errors that occur during the initial match phase may or may not be logged on an entirely arbitrary basis.
We need to adopt a mechanism that will force one rule to be evaluated and fired before any other rule evaluation is undertaken. Fortunately, this is simple to achieve. It requires the use of a very common rule design pattern which I have discussed in previous articles. This pattern involves the use of a ‘sentinel’ value.
Consider the following representation of a very simple initial rule set that sets discount values:
[01]: Assign discount for gold list customers
IF
Transfer.IsClientInGoldList
THEN
Transfer.SetDiscount = 10%
[02]: Assign high value transfer discount for non-gold list customers
IF
Transfer.Value >= 5000
NOT
Transfer.IsClientInGoldList
THEN
Transfer.SetDiscount = 2.5%
Here is how the rule set might look after amending it to use the helper code above:
[01]: Initialise error logging
IF
NOT
RuleEngineExceptionLogger.IsInitialised
THEN
RuleEngineExceptionLogger.SetCompensationHandler(Executor)
Assert RuleEngineExceptionLogger
[02]: Assign discount for gold list customers
IF
AND
RuleEngineExceptionLogger.IsInitialised
Transfer.IsClientInGoldList
THEN
Transfer.SetDiscount = 10%
[03]: Assign high value transfer discount for non-gold list customers
IF
AND
RuleEngineExceptionLogger.IsInitialised
Transfer.Value >= 5000
NOT
Transfer.IsClientInGoldList
THEN
Transfer.SetDiscount = 2.5%
We have introduced a new rule and new conditions into each of the existing rules. The rule set uses the IsInitialised property of the RuleEngineExceptionLogger helper object to obtain a ‘sentinel’ value. If the property returns ‘false’, the new first rule matches. The second and third rules only match if the property returns ‘true’. The ‘initialised’ flag is set to ‘true’ by the call to the SetCompensationHandler method in the first rule.
It is vitally important that we re-assert the RuleEngineExceptionLogger object in the first rule. When the rule set is executed, the engine matches the available facts, including the RuleEngineExceptionLogger object, against each of the conditions in each of the rules. At this stage, only the first rule has a complete match. Once all condition evaluation has been completed, the engine fires the single resulting rule activation. This causes the SetCompensationHandler method to be called and the initialised flag to be set to ‘true’. At this point, the engine would naturally complete its work, having fired all rule activations. However, because the first rule re-asserts the RuleEngineExceptionLogger object, the engine immediately enters a second match-resolve-act cycle. In this second cycle, the first rule no longer matches, but the other two rules may have complete matches, depending on the transfer value and the value returned by the IsClientInGoldList property. Any resulting rule activations for rules two and three are then fired. This, incidentally, is an example of ‘forward chaining’.
If you are unfamiliar with this design pattern, spend a couple of minutes thinking it through. It is really quite easy, and once you have mastered it, it will help to open up a world of new opportunities in regard to rule processing. In this case, it may seem unnecessarily complex to configure logging this way, rather than use the Fact Retriever approach outlined above, and indeed that is the case. However, the reason for describing this approach is to help understand more advanced uses of the compensation feature of the engine.
Anyone familiar with other Rete engines may, if they have read this far, be wondering why I don’t use a common variation of this technique in which the first rule tests for the existence of at least one RuleEngineExceptionLogger fact, and asserts a new instance if none exists. This would be a natural design using most other engines. The reason is that MS BRE does not support Negation nodes. It has no built-in support for negation-as-failure, and you therefore cannot express the necessary rule. I keep drawing attention to this because it is really quite scandalous that MS BRE is missing this common functionality.
The Power of Rule Compensation
The principle reason for executing rule sets within MS BRE is to externalise the rules, separating them from your code, and allowing them to be versioned and deployed orthogonally to the BizTalk artefacts such as orchestrations, maps and schemas. Better than that, you can deploy new versions of your rule sets into a live environment without stopping the application. Rule sets are therefore used to implement policies, which is why, of course, Microsoft provide a ‘Policy’ class to manage execution of specific (or latest) versions of rule sets. Think of your rules as policy statements. For example, a rule might state that “If the value of a transfer is greater than 10,000, indicate that actuarial approval is required”. By externalising the rule, and deploying versioned rule sets, you can easily change the threshold value without disrupting the live environment.
Now consider the following policy statement:
“If the value of a transfer is greater than 10,000, and some unanticipated error occurs during rule processing for that transfer, immediately escalate the problem to the senior account manager (SAM)”.
Using the compensation feature and the Executor engine function, together with a little custom code, you can define this statement directly as a rule within you policy. Alternatively, of course, you could hardwire the rule into an exception handler within a BizTalk orchestration. However, this could mean that, if the threshold value changes, you may need to re-deploy an updated version of your orchestration. You could, of course, make the threshold value configurable, but that would require the value to be stored separate to the policy to which it applies, perhaps within BTSNTSvc.exe.config, or within another rule engine policy. A better design is to define the policy statement as a rule within the same policy to which it applies. If the threshold value changes, or if the business decides to change the policy and no longer automatically escalate these problems, we can control the policy changes by deploying a new version of the relevant rule set.
Implementation of the above policy statement requires a more sophisticated version of the same ‘sentinel’ technique I introduced above. In the following example rule set, I have introduced a ‘Context’ fact that implements a State property as well as methods for adding and removing compensation handlers. The State property can take a number of values. I’ve used strings for these values, but you could consider using an enumeration in real life. The vales represent the notion of discrete states. State transitions are enacted by changing the value and re-asserting the Context object. For each state, there are a number of corresponding rules that evaluate the State property and only run if the engine is in the given state. Hence, the rule set can be understood as a set of groups of rules. Each rule group is associated with a discrete state.
Each rule group, except the group for the terminal state, contains a low priority rule that performs a state transition. This rule must have a lower priority than any other rule in the same state group. The rule will only fire once all other matches for the rules in the group have been found, and the corresponding rule activations have been fired. The rule changes the state of the rule set, and a different rule group is processed. This design pattern allows us to layer a degree of flow control over the set-based pattern matching engine.
In the example rule set, the first rule is used to represent the exception-based policy statement. I am assuming that the rule set is invoked per Transfer (i.e., over a single Transfer object). This is an important assumption. If several Transfers are asserted, and if any one of them has a value greater than or equal to 10,000, then any fault, even if associated with a Transfer whose value is less than 10,000, will be automatically escalated, which is not the intention. You must always think through your rule set logic carefully from a set-based perspective.
Here is the example rule set:
[01.01]: Initialise escalation for errors on high value transfers
IF
AND
Context.State = “started”
Transfer.Value >= 10000
THEN
Context.SetCompensationHandler(Executor, ”escalate to SAM”)
[01.02]: ==== Set state to ‘initialised’ ====
Priority -100
IF
Context.State == “started”
THEN
Context.State = “initialised”
Assert Context
[02.01]: Assign discount for gold list customers
IF
AND
Context.State == “initialised”
Transfer.IsClientInGoldList
THEN
Transfer.SetDiscount = 10%
[02.02]: Assign high value transfer discount for non-gold list customers
IF
AND
Context.State == “initialised”
Transfer.Value >= 5000
NOT
Transfer.IsClientInGoldList
THEN
Transfer.SetDiscount = 2.5%
[02.03]: ==== Set state to ‘discount assigned’ ====
Priority -100
IF
Context.State == “initialised”
THEN
Context.State = “discount assigned”
Assert Context
[03.01]: ==== Tidy up and complete ====
IF
Context.State == “discount assigned”
THEN
Context.RemoveCompensationHandler(Executor, ”escalate to SAM”)
Context.State = “completed”
Rule 01.02 performs a state transition from ‘started’ to ‘initialised’. Rules 02.01 and 02.02 are used to set the discount rate, and rule 02.03 then performs a state transition to ‘discount assigned’. There is one final rule which now matches and fires. This is used to tidy up the engine instance by removing the compensation handler. It also sets the state to ‘completed’ but does not redundantly re-assert the Context object.
I introduced the final rule on the assumption that a Fact Retriever is used to assert the Context object. This allows the engine instance to retain the Context object across rule engine executions, which is, I suppose, very slightly more optimal. Other facts will be retracted automatically before the engine instance is returned to the cache. Each time the rule set is executed, the Fact Retriever will need to set the state to ‘started’ and re-assert the Context object. If you assert the Context object through the Policy object or the Call Rules shape, there may be no need for the final rule.
The SetCompensationHandler and RemoveCompensationHandler methods take a second argument that selects the required compensation handler. This is a little redundant within this simple example. However, you might want to implement this in more complex scenarios. Remember that the CompensationHandlerInfo object contains a callback delegate, and that all .NET delegates can be optionally used as multicast delegates containing an invocation list with multiple delegate entries. You can therefore implement code to add and remove different compensation handlers and have multiple handlers configured at the same time.
There are endless possibilities with regard to the implementation of the compensation handler itself. One simple approach would be to wrap the exception object in an outer exception, assign additional values to the outer exception and then throw the exception back to the orchestration. This approach allows us to continue to use orchestration exception handling whilst retaining the benefits of policies. Another approach might be to write a file to a known receive location or invoke a web service in order to emit an escalation message back into BizTalk. If you prefer, the exception handler could simply do the escalation work itself.
Remember that you can use the user data object. In the example rule set, we will assume that this have been created in the call to SetCompensationHandler. We can implement additional functionality within the Context object to set values on this object. Each time an exception occurs, the user data will be returned to the exception handler, giving us a convenient way to parameterise the handler.
Note that I have named my rules using numeric references. The reason for this is that the rule composer always displays rules in strict alphabetic order using lexical, rather than numeric, ordering for digits. This is a problem when implementing the state transition design pattern because we lose visibility of the rule state groups. However, by adopting a naming convention similar to the above, you can control the order in which rules are displayed, making life a lot easier. I have also used a convention to make it easier to distinguish state transition rules from rules which implement policy statements.
Compensation and Orchestration Design
We have now moved on from the simple exception logging requirements that prompted this article and started to implement general purpose policy-drive exception handling. This approach may encourage some interesting changes to the way you design BizTalk solutions. BizTalk developers are very used to controlling process and workflow though orchestration and this often extends to direct invocation of custom business logic and data access code. Rule processing is typically treated as a adjacent service exploited by orchestrations to compute values which can then be passed back to the orchestration and used to control the flow of a process and the invocation of custom business logic. Consider, then, our example rule set. It exhibits a potentially fatal flaw in relation to the overall design of a BizTalk application. Experienced BizTalk orchestration developers know that the job is, at best, only half finished if their orchestrations only handles the ‘happy’ paths. The ‘unhappy’ paths are vitally important in good process design, and are often the main basis for success criteria. The trouble with rule set compensation is that it only applies to rule processing. If a rule engine is used simply to render some intermediate results which are then handed back to an orchestration and used to govern the process flow and parameterisation of business logic, there is really little point in using rule engine compensation. After all, what does the example rule set suggest? Do we only want to escalate failures to the senior account manager when the rule engine throws an error? What if something goes wrong within another part of the calling orchestration, or within a business logic component invoked directly from our orchestration? At that point, the rule engine compensation handler does not apply. It would be better to allow the engine to throw all errors back to the orchestration and use centralised error handling to decide what to do. Indeed, a centralised error handler could be driven using another rule engine policy.
What BizTalk developers habitually miss is the power of Microsoft’s rule engine thanks to its direct support for .NET. MS BRE only consumes facts in the form of .NET objects (XML, for example, is asserted as an XML DOM document object wrapped in a TypedXmlDocument object), and can (fairly) freely invoke methods and properties on ‘plain old CLR objects’ (POCOs – with apologies to the world of Java). It can do this in both rule conditions and actions. This offers the powerful capability to inject policy directly at the interface between an orchestration and the various business logic and data access components that the orchestration controls. It layers policy directly onto process control, and enriches the context in which custom business logic is invoked.
Consider the following two diagrams. The first illustrates the way in which developers tend to regard and use the rules engine as a means for validating, transforming and rendering data values for subsequent re-use. I’ve termed this ‘policy rendering’. The second diagram illustrates how the engine can be used as a ‘policy injector’ at the orchestration boundary. Both patterns are entirely valid, and may be used side by side. They may even prove to be synergistic. However, the second approach is often overlooked.
The concept of policy injection is well documented. For example, see Microsoft’s Policy Injection Application Block at http://msdn2.microsoft.com/en-us/library/bb410104.aspx. This application block is aligned to the basic ideas of aspect-orientated programming, and provides a pattern for separating crosscutting concerns from the core application logic and applying policies at the level of code member access. For example, we may wish to apply a policy that says ‘if the client code attempts to invoke any public method on the Transfer class, ensure that the security principal has authorisation to do so’.
There is a different emphasis in business rule processing. Business rules apply policy at a higher level. The core concerns are not individual methods or classes, but entire business activities. These are represented at a coarse level by rule sets themselves, and at the fine-grained level by the actions we call when certain rules are fired. They may often be implemented through discrete members of programmatic types, but they represent a much higher-level abstraction. Hence, it would not make much sense to have the equivalent of a transparent proxy (see the application block documentation). Our orchestrations should be concerned primarily with managing process flow and the invocation of business activities such as ‘Transfer Funds’ or ‘Authorise Purchase Order’. They should not concern themselves unduly with the detailed mechanics of business logic implementation. This is one good reason why XLANG/s does not pretend to be a full-featured general purpose programming language.
The crosscutting concerns are business-orientated, rather than technical. This may include concerns analogous to those that the application block addresses, such as authorisation and validation. However, the emphasis is on business, rather than technical requirements. Valid crosscutting concerns might also include things like maximising revenue, streamlining delivery schedules or enforcing regulatory compliance. The intention is to use rules engines to apply business policies to business facts, backed by the inferencing capabilities of the technology, in order to control, monitor and manage the core business activities. Ultimately, at the bottom of the conceptual stack, these activities are represented by the implemented business logic contained in our code.
We can use the rule engine to invoke business logic components, data access code and other functionality directly. By moving the invocation of this code away from orchestration expression shapes and into rule sets we start to open up a many new possibilities. True, the rule engine does not provide an equivalent environment to orchestration. For example, an orchestration can enlist transactional components (built using .NET Enterprise Services) directly into its own internal atomic transactions. The rule engine does not support this capability. You must also be very careful with regards to serialisation of custom components. The rules engine does not support serialisation of its state or the facts contained in working memory. If you build custom logic into serialisable classes, you can persist the state associated with your custom code inside de-hydrated orchestrations as long as you invoke your code directly from an orchestration using orchestration variables. This is not the case if you invoke the same code from a rule set. I am not advocating a wholesale change in design from policy rendering to policy injection, but simply highlighting the opportunity to exploit the available toolset in more sophisticated ways where appropriate.
The policy injection pattern makes much more sense of the compensation handler feature of the rules engine. For our simple example rule set, there we can increase the value of the ‘escalation on error’ rule if the rule set directly governs a coarse-grained chunk of work. Orchestration can be used, as originally intended, to govern and manage the overall process flow. To process the Transfers in a policy-driven fashion, the orchestration can invoke the rule set and let it do the fine-grained work in a policy-driven fashion. When the rule set execution is complete, the orchestration can inspect the results in order to decide how to progress to the next coarse-grained activity.
Again, let me stress that I am not advocating policy injection as some sort of panacea. I could, if I was inclined, think up a dozen good reasons why you might choose to avoid this approach. As I see it, the main deciding factor has to do with the extent to which discrete, fine-grained activities must be governed and driven by business policies. Careful judgement is needed to decide the best design approach on a case-by-case basis. One issue you will face has to do with the lack of deep integration between BizTalk Server and the rules engine. The engine has no support for the internal BizTalk message bus (i.e., the Message Agent and Message Box). The policy injection pattern would be more useful if MS BRE provided functionality similar to the ‘Send’ and ‘Send Orchestration’ shapes, or even (a very ambitious suggestion) the ‘Receive’ shape. I’m not convinced, though, of the feasibility of integrating BizTalk’s subscription and correlation models directly with a set-based pattern matching engine.
Let me finish by pointing out that we are now only half a step away from some sophisticated design patterns involving dependency injection and inversion of control. There is much more I could write on this whole subject, including investigating how, by handing the Executor off to custom code invoked from within a running rule-set, we have even more powerful (but potentially dangerous) opportunities to implement advanced patterns. This article is long enough already, and I will resist the temptation to explore these avenues further on this occasion.