Charles Young

Peter Lin published a set of simple Jess/Clips tests designed to show scalability characteristics in various scenarios. These can be downloaded from http://dingo.sourceforge.net/rete-benchmark.zip. I undertook to replicate these tests in the Microsoft Business Rules Engine for comparison, and to publish the results. There were five tests in all. However, one of these tests (Ordered Facts) was not repeatable in MS-BRE. In contrast to Jess and Clips, MS-BRE does not employ a LISP-like approach, and has no equivalent concept of ordered facts. Rather, facts are always asserted as .NET objects (using specialised classes as required to bind to XML documents or ADO.NET datasets), and facts in MS-BRE are therefore somewhat similar to the Jess/Clips concept of unordered facts.

I encountered two difficulties while replicating the tests. The first is that some of the rules files provided by Peter for the Mixed-Set test appear it be incorrect. Rule files rules5k-mix.clp to rules10k-mix.clp, inclusive, appear to use the ‘object1’ fact in every rule, whereas they should, I believe, be using the ‘object2’ fact for rules 1000 and greater. I ran the provided rules through Jess, and got very different results to Peter. I then amended the rules to what I believe they should be, and re-ran the tests. This time I got similar results to Peter’s. I have therefore used the amended rules here.

The second problem concerns the Unique-Object test. The rule files provided by Peter appear to be correct, but every time I run this through Jess on my local machine, I get significantly different results to Peter’s. This is shown in the graph below:

I wondered if perhaps the original test had been conducted with varying numbers of facts. However, Peter’s results indicate that exactly 1000 rules fired for each rule set size, which implies that the same 1000 facts were submitted for each run. Certainly I would expect the Rete algorithm to provide results similar to the one I obtained, rather than those provided by Peter. For the purposes of this comparison, I have used the rules and data provided by Peter, but have compared the MS-BRE results to those obtained using Jess on my machine.

I have provided further details of how I translated the rules in MS-BRL in Appendix A I have provided the full results in Appendix B. The results are summarised below. Please note that, again in order to avoid unhelpful comparison or raw performance, I have adjusted the ‘y’ axis for the first two tests (1K-Facts and 1K Facts Fire Each) so that a direct comparison of scaling characteristics can more easily be made. For other tests, I have shown the graphs using actual times.

Results

1K Facts

1K Facts Fire Each

Unique Object

(NB. Jess times are mine, rather than Peter’s – see above)

Mixed Set

Conclusions

The results suggest that, within the limitations of the tests published by Peter, the Microsoft Business Rules Engine exhibits similar scaling characteristics to Jess. This, in turn, may provide some evidence to support the claim that Microsoft’s engine implements the Rete algorithm correctly.

Appendix A

Rule Translation

In order to conduct the tests, it was necessary to translate the Jess/CLIPS rules into equivalent Microsoft Business Rule Language. This appendix provides indicates the form of the MS-BRL rules. The same form was used across all rules, although the names of rules and ‘object’ facts varied, as did the value of the attr1 slot.

Examples of the rules are shown below:

Jess

    (defrule rule0
        (object1
        (attr1 "0")
      =>
        (printout t "rule0 fired")
    )

• MS-BRE

    <rule name="rule0" priority="0" active="true">
      <if>
        <compare operator="equal">
          <lhs>
            <function>
              <classmember classref="object1"
                           member="get_Attr1"
                           sideeffects="false" />
            </function>
          </lhs>
          <rhs>
            <constant>
              <string>0</string>
            </constant>
          </rhs>
        </compare>
      </if>
      <then>
        <function>
          <classmember classref="object1"
                       member="PrintOut"
                       sideeffects="true">
            <argument>
              <constant>
                <string>rule0fired</string>
              </constant>
            </argument>
          </classmember>
        </function>
      </then>
    </rule>

N.B., The MS-BRL rule is more closely equivalent to the following Jess rule:

    (defrule rule0
        (object1
        (attr1 ?x&:(eq ?x "0")))
      =>
        (printout t "rule0 fired")
    )

MS-BRE does not implement a LISP-like approach, and has no equivalent to Jess’ ability to look up facts in the knowledge base. Note that the Jess rule used by Peter is significantly more efficient that the alternative shown here.

Appendix B

Detailed Results

As I have done elsewhere, I must emphasise that the purpose of these tests was to investigate the scalability characteristics of the three engines in relation to Microsoft's original test results for MS-BRE, and not to provide evidence of performance in terms of speed. All times in ms.

1K Facts

1K Facts Fire Each

Unique Object

Mixed Set

All tests were conducted on the same machine. The specifications are provided below.

The .NET code ran under .Net Framework 1.1.4322