A couple of days ago I found myself, yet again, introducing another BizTalk developer to the delights of using XPath with XML facts in Microsoft's rule engine. I've begun to lose count of the number of times I have been asked to troubleshoot issues with the rules engine only to find that the problem is a lack of understanding of how XML facts are manipulated in rules. It's about time I wrote something about this problem. The issues are not complex, but need to be explained clearly.
It’s a problem
If you have used the Microsoft Business Rules Engine (MS BRE) from within BizTalk, you have probably written rules that act on XML messages. The XML content of a message part (or any XML document or node) can be 'asserted' to the rule engine as a 'fact'. Consider the following very simplistic XML document:
<MyMessage>
<AStringField>Hello, world!></AStringField>
<ANumericField>15</ANumericField>
</MyMessage>
It can't get much more basic than this. Note that, for clarity, I haven't used namespaces. I will also assume that you have an XSD schema for this XML.
Let’s consider the following scenario. In the Rules Composer, you create a new policy and add a rule. You reference the schema in the 'Facts Explorer' pane and then create the following rule by dragging the 'ANumericField' element to the first placeholder in a 'Greater Than' predicate. I'll express the rule in Rule Composer-like pseudo code:
IF
mySchema:/MyMessage/ANumericField is greater than 10
THEN mySchema:/MyMessage/ANumericField = 10
The action in the THEN section is not relevant here.
So far, so good. The policy is published and you then invoke this policy over a message in BizTalk, probably using the Call Rules shape. Now comes the problem. You discover, during testing, that some messages don't have an 'ANumericField' element. Your orchestration faults, complaining that:
Field "*[local-name()='ANumericField' and namespace-uri()='']" does not exist in XML document "mySchema", selector "/MyMessage".
Too true. There appears to be a straightforward solution, however. The MS BRE has an 'exists' predicate which is designed specifically for use with XML facts. The predicate applies an XPath to the XML, and returns 'true' if this returns any nodes. The rule is therefore amended to read:
IF
AND /*[local-name()='MyMessage' and namespace-uri()=''/*[local-name()='ANumericField' and namespace-uri()=''] exists in mySchema
mySchema:/MyMessage/ANumericField is greater than 10
THEN mySchema:/MyMessage/ANumericField = 10
This is what you see if you create the rule through drag and drop. Because we are not using namespaces, we could reset the XPath argument and then type in a less verbose XPath directly:
IF
AND
/MyMessage/ANumericField exists in mySchema
mySchema:/MyMessage/ANumericField is greater than 10
THEN
mySchema:/MyMessage/ANumericField = 10
That’s a bit more readable. However, avoiding namespaces is generally not good practice in the integration world. You could, however, create and use a vocabulary item for the 'ANumericField' element to make the rule even more readable.
You have a new version of our rule which looks like it should work. You run it and...disaster. You get exactly the same error message. Two agonising hours later, you have been reduced to a pathetic figure, sitting in a dark corner muttering incomprehensibly to yourself, and totally unhinged by the MS BRE.
In an attempt at rehabilitation, let me explain, step by step, what you must understand to solve this kind of problem. It is actually quite straight-forward. All that is needed is an explanation, and a little guidance. First, consider the following two points:
It's a pattern-matching engine!
Perhaps the single most important concept to grasp when learning to use the MS BRE is that it is a pattern-matching engine. Rules contain conditions, and each condition is a pattern. The engine works by matching each condition to each relevant fact asserted to the engine. The engine searches for sets of facts that match all the conditions with a rule. Whenever it finds such a set, it creates an instance of the rule on a prioritised queue called the Agenda, and provides this instance with the set of facts. The actions in the rule can then be executed over the data contained in the facts. There is more to it than that, but the important thing here is to understand that MS BRE is a pattern matching engine.
Pattern matching is not unusual. XPath is a pattern-matching approach used to address XML documents, and is used extensively in XSLT. In turn, XSLT is a pattern-matching transformation language for XML. Regular Expressions offer another form of pattern matching.
Rules in MS BRE are very similar to another pattern matching language with which many developers are very familiar. You can think of a rule as being rather like a SQL SELECT statement with behaviour (the rule's actions). In SQL, a single SELECT statement may match zero or more matching records. 'Facts' in MS BRE are really a relational concept. Each fact is treated much like a data row in a Relational Database table. MS BRE doesn't have tables as such, but it does understand that each fact is of a specific type, and can therefore match all the facts of a specific type against the relevant conditions contained in the rules.
Short-circuiting
MS BRE is a pattern matching engine which tests conditions against entire sets of data. This is quite different to procedural code, and the approach to short-circuiting is also different. What is more, although it is entirely logical, it can appear confusing and arbitrary to developers who are used to short-circuiting within procedural code. MS BRE will sometimes short-circuit, but often it won't.
To understand why, consider a fact (a piece of data) that is being processed by the engine. The engine passes the fact through an in-memory node network (the Rete). Each node performs conditional evaluation of the fact, and if the fact passes a test, it is handed on to the next node. If it fails the test, it is not passed on. The fact is first passed to nodes that test only a single type of fact. Then, if required, it is later passed to nodes that perform joins on other fact types.
Let's consider a case where fact 1, of type A, is being evaluated against a simple rule that contains two conditions. Each condition tests a different field of type A. The fact arrives at node (i) which performs the first evaluation. If the fact passes, node (i) passes the fact to node (ii) which performs the second test. However, if the fact fails the test in node (i), it is not passed to node (ii). Node (i) has effectively 'short-circuited' the evaluation.
Using two nodes in this way is actually a little inefficient. In this example, MS BRE will create a single node containing two tests. In order to achieve the same effect as having two nodes, the first condition will short-circuit the second.
The important point here is that the short-circuiting is a direct consequence of the node-based approach to rule processing used internally by MS BRE. If two facts are asserted to the engine, and each fact is of a different type, they will pass through completely different node paths, Note, however, that they may 'meet' in later nodes if the first fact type is joined on the second. Short circuiting only ever applies in situations where you have multiple conditions in a rule that test the same fact type. This is entirely different to the model that many developers expect. If they create a rule with two conditions, each of which tests a different fact type, and the first condition evaluates to 'false', they naturally, but incorrectly, expect short-circuiting to occur.
There is further consideration. The 'short-circuiting' I have described only applies to situations where conditions are grouped using the AND connective (a conjunction). It never applies to disjoint conditions. Disjoints are created using the OR connective. If your rule contains two conditional tests for the same fact type, and those conditions are ORed, MS BRE will never short-circuit. In effect, a rule that uses OR connectives is treated as if it is really composed of multiple rules. This is the normal approach taken almost universally by similar engines. In many engines, it can lead to a duplicate 'firings' of the same rule where developers expect (and want) just one. However, MS BRE contains additional code to prevent this.
A bad solution
Developers often persevere in trying to use the Exists predicate as part of a solution to the type of problem I described above. One approach I’ve seen is to create rules that contain a single condition using the 'exists' predicate. The actions for such rules are written to set some flag. The flag is typically set either in the XML itself (possibly requiring the introduction of a new element or attribute into the schema for this purpose), or in some.NET 'helper' object asserted to the engine for this purpose. The flag captures the result of evaluating the single 'exists' condition, and we can then use it to select a second rule that tests the value of the existing element or attribute. You have to assert, or re-assert, the fact containing the flag in order to start a new match cycle. The engine can then match on the second rule and fire instances for each match found.
If this sounds complex enough, be aware that you need to jump through another hoop. When the ‘exists’ rule asserts the fact containing the flag, you must ensure that this same rule no longer matches. To do this, you need to ensure that your flag has a default value, and that you match that value in the 'exists' rule. The flag must be set to a non-default value in order to match the second rule. In pseudo code, a simple policy using a new ‘existsNumericFields’ attribute might look something like this:
IF
AND
mySchema:/MyMessage/@existsNumericFields is equal to '0'
/MyMessage/ANumericField exists in mySchema
THEN
mySchema:/MyMessage/@existsNumericFields = '1'
assert mySchema
IF
AND
mySchema:/MyMessage/@existsNumericFields is equal to '1'
mySchema:/MyMessage/ANumericField is greater than 10
THEN
mySchema:/MyMessage/ANumericField = 10
For all kinds of reasons, this is not a very good approach. First, it is contrived. We have to introduce an additional flag in order to capture the results of the 'exists' test. This flag has nothing much to do with our business rules. It functions purely as a mechanism to get the policy to work as we wish. Another problem is that we must ensure that the flag exists and is set to a default value. This is fairly natural if you use, say, a Boolean property on a .NET object. However, our example uses an additional attribute introduced into the XML. We have to ensure that this attribute exists in every XML message or document we assert to the engine, and we have to make sure that the value is set to '0'. Note how the semantics of this attribute are unclear. Before the XML is asserted to the engine, the attribute appears to state that there are no 'ANumericField' elements, even if the XML does actually contain such fields. Yet another problem is that the approach is inefficient. The engine is forced to perform two match-resolve-act cycles where, before, it performed just one.
Despite several drawbacks, it happens that there are good reasons why this general type of approach can often be useful. Unlike many similar engines, the MS BRE has no support for what the logicians call 'existential quantification' and cannot, therefore, directly perform weak negation ('negation-as-failure'). Using flags in this way can often be used to work around this problem. This issue is beyond the scope of this article, so I have written more about this. You can find the article at http://geekswithblogs.net/cyoung/articles/90100.aspx.
XML isn't relational
Before discussing a better solution to our problem, consider an obvious issue. XML is hierarchical. As we have seen, the MS BRE is relational in nature. There is, therefore, a fundamental impedance between XML and the MS BRE. Microsoft has some considerable expertise, of course, in dealing with this impedance, and implements a number of approaches which map between hierarchical and relational data. SQL Server 2005, for example, has deep support for XML, and since its inception, ADO.NET has unified the two data models within its in-memory data sets which can equally be viewed as a collection of relational tables or as an XML document.
MS BRE does not offer any great sophistication here. In order to map the hierarchical XML model onto MS BRE's relational model, you use XPaths to select sets of nodes. Each node is treated as a 'tuple' (the equivalent of a row of data in a relational table). XPaths are used to map from XML into sets of facts which the engine then acts on. This is an elaboration of the normal semantic of 'assertion'. When XML is asserted to the MS BRE, the XML document is given special treatment. In a sense, it is not really a 'fact' in the normal sense. It is a hierarchical collection, or source, of facts. XPaths are used to select the facts from this collection as required.
This is entirely logical, and there aren't many alternatives. A more sophisticated approach might support XQuery, rather than XPath. XPath is not really a query language, but it shares many similarities with query languages, and provides selection and filtering features. This is actually a source of confusion in MS BRE because, as I suggested above, rules are, themselves, a form of query language similar to SQL. In order to apply rules to XML, we have to layer XPaths on top of our rules. We are using a (semi) query language directly on top of another query language!
Using XPaths
Once you have grasped the role of XPaths and their interaction with rules, all that remains is to understand how to manipulate XPaths in MS BRE. This is very straightforward. However, it is not obvious, and until the mechanisms are pointed out, you might well not understand what facilities are available to you. When you reference a schema in the 'Facts Explorer', the Rules Composer automatically generates XPaths to select the 'records' and 'fields' in the schema. Note, in passing, how the BizTalk XML schema editor itself layers a schema model on top of XSD that has some similarity to the relational model. Records and fields have no direct analogue in XSD, and must be mapped onto definitions of elements and attributes. A 'field', for example, may map onto either an element or an attribute, and XPaths are used to perform this mapping. Note that Microsoft’s XML schemas are not relational. Records can contain nested records in a hierarchical fashion.
XPaths are used to select node sets. In order to map a node set onto a set of facts in MS BRE, MS BRE defines an XPath referred to as the 'XPath Selector'. You will find it in the 'Properties' pane at the bottom left corner of the Rules Composer. Don't misunderstand what is going on here. The properties listed in this pane are just a much a part of your rule set as the rules you create on the right hand side of the composer. The XPath Selector is layered on top of your rules, and can be considered as an extension to your rules. This is logical, but not pretty. It can also be confusing.
If you select a field in your XML Schema, you will see a second XPath in the Properties pane. This is termed the 'XPath Field'. This XPath is incomplete. It is not rooted, and is designed to be used to extend the XPath Selector in order to select individual data values. It is important to distinguish between the XPath Selector and XPath Fields. XPath Selector are used to select node sets. Each node is treated as a fact. XPath Fields are then used to select individual values from those facts.
The two properties are auto-generated, but are editable. This is really the central point of this article. By taking control of your XPath Selectors and Fields, you can solve a whole host of apparent 'problems' when applying MS BRE rules to XML Unfortunately, however, the Rules Composer does not make this very easy. XPaths tend to get very verbose, especially when your XML uses namespaces. You have to edit these XPaths directly in the Property pane or within a wizard. There is no specialised editor, and no facilities for testing your XPaths against sample XML. There are no visual prompts that allow rule developers to understand the association between XPath Selectors and Fields and their relation to your rules.
Rather like the ‘exists’ predicate, the Rule Composer displays versions of your XPaths. However, it is abbreviated. Abbreviation helps to make things a bit more readable, but rules can rapidly become difficult to decipher. You may wish to create vocabulary definitions and map them to 'records' and 'fields' in your XML schema. These definitions contain an XPath Selector and XPath Field which you can edit through the Vocabulary Definition Wizard. You are not allowed to edit it directly in the Properties list. Using vocabulary definitions allows you to create multiple mappings to the same nodes in your XML. For example, you might sometimes want to apply a filter in an XPath in order to further constrain node selection, and sometimes select the same nodes without the filter. You can create multiple vocabulary definitions in order to do this.
Solving the Problem – First Approach
Now we have introduced XPath Selectors and Fields, let's see how we can use them to solve the original problem. The issue was that sometimes our XML does not contain a 'ANumericField' element. When MS BRE tries to match this XML to the rule condition, you get an error because the condition assumes that this element always exists. To understand why, we need to inspect the auto-generated XPaths created by MS BRE.
If you create a simple schema for the XML example, and reference it in the Rules Composer 'Facts Explorer' pane, you will be able to view the auto-generated XPaths for the 'ANumericField' field. The XPaths will look something like the following:
XPath Selector: /*[local-name()='MyMessage' and namespace-uri()='']
XPath Field: *[local-name()='ANumericField' and namespace-uri()='']
‘ANumericField’ is defined as a field within a ‘MyMessage’ fact. The XPath Selector is used to select all ‘MyMessage’ nodes in the XML (there is just one, in this case), and the XPath Field is used to address the ‘ANumericField’ element. Note a restriction here. The XPath Selector is used to select a node set. The XPath Field is used to select a single node. If you have programmed with the XML DOM, you will be very familiar with these two approaches. If your field is a repeating element, the XPath Field will select the first element only.
Note that, from one viewpoint, the XPath Selector can be thought of as a type definition. It defines the 'type' of fact we are interested in, and is used to select facts of that type from an XML document. Each node returned in a selection represents a single fact of the 'type' defined by the XPath Selector.
You can edit the XPath Selector in order to select exactly which facts you want to assert to the engine. Sometime, you have to make corresponding changes to the XPath Field. There may also be occasions when you want to edit the XPath Field to select a specific element from a group of repeating elements. Indeed, it is perfectly possible to edit these XPaths so that the engine selects entirely different nodes to those which your XML schema fields and records are bound! There is no direct binding between the underlying XSD and your rule set. XML schemas are used in the Rule Composer as a convenient means of auto-generating XPaths. Your MS BRE rules are then annotated with these XPaths in order to select data from XML documents. You should generally avoid, however, breaking the link between your rules and the underlying XSD. If you do this, you are sowing the seeds of confusion.
Note, in passing, that there are occasions when you may need to change the value of the Type property to correspond with changes you make to the XPath Field. The Type property controls how the textual content of an XML element or field is converted into .NET data.
One way to solve the example problem is to edit your XPath Selector to include a filter. Filters in XPath are contained in square brackets. Add a simple filter to test for a named child element. Your XPath Selector will look something like:
XPath Selector: /*[local-name()='MyMessage' and namespace-uri()=''][./*[local-name()='ANumericField' and namespace-uri()='']]
Because the example XML does not use namespaces, you could simplify this further:
XPath Selector: /MyMessage[ANumericField]
Here is a another approach which you may prefer. Edit the XPath Selector to select ‘ANumericField’ elements as facts. You will then need to use the ‘.’ Operator in XPath to select the text content of your facts as the field.
XPath Selector: /*[local-name()='MyMessage' and namespace-uri()=''] /*[local-name()='ANumericField' and namespace-uri()='']
XPath Field: .
Again, you could simplify the XPath Selector to read:
XPath Selector: /MyMessage/ANumericField
With these changes, you no longer need to include the 'exists' condition in your rule set. Retain the single condition that tests the value of ANumericField. Your new XPath Selector will ensure that facts are asserted only for existing ‘ANumericField’elements. There will be no facts asserted where this element does not exist, and hence the engine will not encounter an error condition.
Solving the Problem – Second Approach
There is a second approach we can take. For simple problems, this may be the more desirable approach because it allows us to continue to use the ‘exists’ predicate and helps to maintain the clarity of rules. In fact, it is not really a second approach at all, but simply the application of the first approach together with an understanding of the ‘short-circuiting’ mechanism in MS BRE which I described above.
The reason that the ‘exists’ predicate so often proves difficult is that we don’t get the short-circuiting behaviour we expect. We combine a condition that tests for the existence of nodes within our XML with one or more additional conditions that test ‘field’ values for those nodes. The first condition evaluates to false, but does not short-circuit the evaluation of the other conditions. The engine attempt to evaluate those conditions against non-existent nodes, and throws a run-time exception.
We saw that the engine will short-circuit in very specific circumstances. If a rule contains multiple conditional tests that a) evaluate the same fact type and b) do not perform joins on other fact types, the engine will generally short-circuit for this group of conditions. Our problem, therefore, is that the condition that uses the ‘exists’ predicate is using a different fact type to the other relevant conditions.
This is a very common problem, but armed with our knowledge of selectors and fields, we can often solve it. To do so, we need simply to ensure that the XPath Selector for the node that we use as the ‘xml document’ argument to the ‘exists’ predicate is identical to that used on other conditions. Unfortunately, when you drag and drop in the Rules Composer, this is rarely the case. Even if you drag and drop an inner node from your schema, you find that, as soon as you drag and drop another node to the ‘xpath’ argument, the composer automatically sets the ‘xml document’ argument to the root.
The answer is to manually type in your XPath for the ‘xpath’ argument. When you do this, the Rules Composer does not change the ‘xml document’ argument. Your XPath needs to be specified relative to the node you are using for ‘xml document’. At runtime the rule engine will assert each of these nodes as a separate fact, and the rule will be evaluated against each node in turn. Because the other relevant conditional tests are defined against the same fact type, short-circuiting will ensure that you don’t get runtime exceptions.
In the following example, the rule is created by typing 'ANumericField' into the left argument of the 'exists' predicate, and dragging and dropping the root node of the document to the right argument.
IF
AND
ANumericField exists in mySchema:/MyMessage
mySchema:/MyMessage/ANumericField is greater than 10
THEN
mySchema:/MyMessage/ANumericField = 10
The advantage of this approach is that it allows you to use ‘exists’ predicate and also often reduces the verbosity of your XPaths. The disadvantage is that the approach is ‘brittle’. If someone edits your rule at a later date using the rule composer, they may drag and drop onto the left argument of the 'exists' parameter, re-introducing the original problem without being aware of the consequences of their actions.
Conclusion
Controlling your XPath Selectors and XPath Fields is the key to effectively applying rules to XML in MS BRE. Remember that you can also assert other types of data. MS BRE understands .NET objects, as well as relational data sources such as ADO.NET DataTables and DataRows and connections to live databases. Incidentally, MS BRE supports data connections to SQL Server databases and also to OLE-DB data sources. Unfortunately, however, the Rule Composer supports SQL Server connections only.
XPaths are used to map the hierarchical data model of an XML document onto the relational model used by MS BRE. This mapping layer is vital, and is fully controllable by the rule developer. In order to use MS BRE effectively over XML data, you need to ensure you understand XPath.