This article supplements an earlier article, posted a couple of weeks ago (http://geekswithblogs.net/cyoung/articles/90102.aspx), on issues concerning the use of XPaths to map between the hierarchical data structure of an XML document and the relational view of 'facts' in the Microsoft Business Rules Engine. As I suggested in the introduction to that article, I have repeatedly been asked to troubleshoot MS BRE issues which turn out to be related to the use of XPaths. Earlier this week, I received another request for help. The symptoms and solution to the problem are worth recording because they provide greater insight into this topic.
A slightly simplified version of the symptoms are as follows (I have 'smoothed' some ragged edges to make it easier to understand the essential problem). A simple 'decision logic' policy contains seven rules. An XML test document contains seven ‘dataRow’ elements containing a number of nested elements. When the XML test document is asserted to the engine, the expectation was that each rule would be evaluated against each ‘dataRow’ element. This would result in 7 x 7 (49) evaluations.
The conditions of each rule contain a call to a custom 'predicate' method - i.e., some custom method that returns a Boolean value, and which is used to evaluate the condition. Each rule uses a different method. The value of an element nested in the ‘dataRow’ fact currently being evaluated is passed as a single parameter to a method, and each method is implemented to perform a look-up on a SQL Server database.
So far, so good. There is something almost Biblical about the figures here, and even more so when it was learned, by tracking activity against the database, that instead of performing 7 x 7 evaluations, the policy performed 7 x 7 x 7 (343) evaluations. Perhaps this indicates we should forgive Microsoft, not 7 times, but 7 x 7 x 7 times (OK, it's 70 x 7 (490) in the Bible, but who’s counting?).
I was asked to comment on what was going on. My response was to ask to see the policy, and sure enough, the answer soon became apparent.
Each condition, except one, contained two (or more) conditions. One condition invoked the custom predicate method. Another checked the value of another element nested in a ‘dataRow’ element. Don't worry about the rule that has just one condition (a call to a predicate method). We will come back to that later. Here is an example of one of these rules:
IF
AND
TxGroup:/TxGroup/dataRow/hasActivity is equal to Y
NOT
ContactValidationHelper.IsActivityValid( Activity )
THEN
SetErrorCode 10002
SetErrorMessage Activity is Invalid
SetErrorCategory 1
SetExternalRef TxGroup:/TxGroup/dataRow/externalReference
SetExternalRef TxGroup:/TxGroup/dataRow/batchNumber
SetExternalRef TxGroup:/TxGroup/dataRow/transactionNumber
ErrorCollection.AddCopy( ValidErrror )
The rule developer had correctly reasoned that, just because you have two conditions in each rule testing different values contained in the same fact, there was no reason for seeing the 73 evaluations when only 72 were expected. He could not understand why he was getting the observed behaviour. The answer lay in the use of XPath Selectors.
A vocabulary had been created with (you've guessed it) 7 definitions. Each definition referred to a different element nested in the ‘dataRow’ element. No doubt in order to avoid problems where elements are missing (each nested element was marked as 'minOccurs="0"' in the XSD schema), the XPath Selector property had amended to contain a filter. This meant that the XPath Selector would only select ‘dataRow’ elements that contained the element to be tested. So far, so good. The vocabulary definitions had been used to define the values passed to the predicate methods used in each rule.
The trouble was that each rule (except one) contained an additional condition testing the value of another nested element. For these conditions, the developer had simply dragged and dropped the 'field' from the schema without making any changes to the XPath Selector property value. This meant that there were two distinct XPath Selector property values associated with each rule. These were something like the following (I've omitted any namespaces for clarity):
vocabulary: \\TXGroup\dataRow[activity]
schema: \\TXGroup\dataRow
It's easy to understand why the rule developer expected each rule to be evaluated just once against each ‘dataRow’. They naturally expected that MS BRE would realise their intention. However, this is not the case, and for a quite logical reason. The XPath Selector property is used to select a node set from an XML document. Each node is then asserted to the working memory of the engine as an individual fact. Additional XPath Field properties are used to select values in relation to these nodes. In this way, developers map from the hierarchical world of XML to the relations world of the MS BRE.
The issue here is that we have two distinct XPath Selectors. They are not equal. Therefore there is no guarantee that the node sets they produce will contain an identical set of nodes. It is true that the resulting node sets will contain only ‘dataRow’ elements in both cases, but, as a direct consequence of mapping from XML to relational tuples, the engine cannot treat these XPath Selectors as designating the same facts. As I said in the previous article, you can think of the XPath Selector as a kind of type specifier, specifying the type of fact selected from the XML. If one XPath Selector is different to another, the engine will see this as two distinct and different fact types.
If you have grasped that, you will probably have realised why so many evaluations were being done. For each rule, we have two conditions, each testing a different fact type. Yes, we know that both fact types map onto the same ‘dataRow’ elements in the XML. However, the engine cannot afford to make any assumptions. Because each mapping can result in different membership of the node set it produces, the engine must act on two distinct nodes sets, and not one. Obvious really.
MS BRE is a pattern-matching engine, and the conditions of each rule are rather like a SQL Select statement. Each node set can be regarded as equivalent to a table in a SQL database. Put this together, and you can see that, in effect, each rule defines a join between the two node sets. There is nothing to constrain this join, so the engine performs what the relational guys call a 'Cartesian product' We have 7 ‘dataRow’ elements, each of which happens to appear in both the node sets. That means there are 72 unique combinations of two facts of different types. With seven rules, this neatly gives us 73 evaluations in total.
Or does it? What about the rule with only one condition? Surely this undermines my maths. Well, at first glance it appears that we should have less than 73 evaluations, but in fact there was another feature of the rule set which I need to explain. This feature exploited the high-level nature of rule expression in Microsoft's Rule Composer (and MS Business Rule Language) which, whilst making it easier to express rules, can make MS BRE rules harder to troubleshoot than in many other rules engines.
Each rule had a very similar action list containing several actions (you won't believe it, but each one had exactly 7 actions!). None of the rules performed any asserts, updates or retracts, so there was no forward-chaining. However, each rule contained three actions, each of which invoked a different custom helper method. A single value was passed as a parameter to each of these three method calls, and this value was obtained from a nested element of a ‘dataRow’. The developer had used the schema, rather than a vocabulary item, and so the XPath Selectors for these values did not contain any additional filter. Indeed, in all the rules except the one with a single condition, the value of the XPath Selector was always identical to the XPath Selector used in one of the two rule conditions.
As I said before, the conditions in any one rule are rather like a SQL Select statement. They project, select and join. This results in the equivalent of a SQL data set. Internally the engine creates collections of tokens where each token can be regarded as the equivalent of a single row in a data set. A token contains a linked list of Working Memory Elements (WMEs), where each WME provides access to data projected and selected from a single 'fact'. It is this data that is made available to the actions in a rule. When a rule 'fires' it performs actions over the data contained in a single token.
If our action list refers to values of nested elements in ‘dataRow’ elements, then these values must be selected in order to provide data to the action. In other words, we must have additional rule conditions that select these values. In most rules engines, rule developers have to understand this, and are required to add additional conditions explicitly to each rule to ensure that all the required data is selected. However, in MS BRE, when a data value is only used within an action list, and is not otherwise evaluated, these additional conditions are inferred, and are not defined explicitly. This makes rules in MS BRE a little easier to define and understand, but, as you can see, hides the true state of affairs, and therefore makes rules harder to troubleshoot.
Back to our rule with the single condition. This one condition used an XPath Selector from a vocabulary definition, and contained an additional filter. However, the actions used a different XPath Selector property value without the filter. These actions resulted in implicit conditions within the rule, and hence even this rule contains two distinct XPath Selector property values. The glorious symmetry of the example is preserved and we get 73 evaluations.
I have one issue to report with MS BRE’s approach to XPath Selectors. It is fairly minor, but worth noting. There are not that many ways in which you can express logically identical XPaths in two different forms, but here is an obvious one using an axis:
\\TXGroup\dataRow
\\TXGroup\child::dataRow
These two XPaths are guaranteed to select identical node sets from the same XML. They are logically identical. Unfortunately, MS BRE is not smart enough to realise this, and will treat them as two different XPath Selectors. That is really not so bad, as most developers rarely use the ‘child’ axis. However, here is a much more pernicious example:
\\*[local-name()=’TXGroup’ and namespace-uri=’http://tempuri.org’]
\\*[local-name()=’TXGroup’ and namespace-uri=’http://tempuri.org’]
Blink, and you will miss it. The second XPath has an additional space before the ‘and’. Yes, you’ve guessed it. MS BRE will conclude that these specify different fact ‘types’. You need to be ruthlessly consistent in your use of XPath Selectors. This example shows how they can be a potential source of very opaque bugs.
No wonder people find processing XML facts so hard in MS BRE. If you have a choice in the matter, I seriously suggest that you consider representing facts in another form. I quite like using .NET objects to represent facts. Although there is a mapping layer between the OO world and the relational viewpoint of the engine, this is far more implicit, and does not require layering XPaths, or some equivalent, on top of an existing query language. A great benefit of using .NET objects is that they can be ‘smart’. The developer has more scope for deciding the most appropriate way to implement their business logic, and can, where appropriate, hard-wire logic into their facts rather than jump through hoops trying to invoke custom logic through rules. MS BRE is designed primarily to apply rule processing directly to the data in your business logic tier, and using .NET objects is a great way to control and exploit beneficial tradeoffs when bringing logic to bear on data. Of course, there is the small issue (and, in my experience, it has proved a surprisingly small issue in BizTalk development) of MS BRE’s approach to managing truth maintenance when using .NET objects, but I will keep that for a future blog article.