- Business Rule Engine

BizTalk 2006: Deploy the Policies with command line (BTSTask)

Leonid Ganeline — Tue, 10 Apr 2007 15:41:00 GMT

It is not so strightforward as it suppose to be.

Below are the bat files with comments:

@rem ----------- Deploy.Policies.bat ---------------------------------------------
@call Settings.bat

@ECHO.
@ECHO. Deploy Policies:
@ECHO. ======================================================================
@ECHO.

@Time /T
@ECHO 1) Create the ResourceSpec.xml file.
@rem BTSTask ListApp /ApplicationName:%ApplicationName% /Server:%BizTalkDataBasesServer% /Database:%BizTalkConfigurationDataBase% /ResourceSpec:"%CurrentBuildPath%\ResourceSpec.xml

@Echo 2) Then edit it to remain only Policy recources.

@Echo 3) Export .xml to .msi
@rem BTSTask ExportApp /ApplicationName:%ApplicationName% /Package:"%CurrentBuildPath%\Policies.msi" /ResourceSpec:"%CurrentBuildPath%\ResourceSpec.xml" /Server:%BizTalkDataBasesServer% /Database:%BizTalkConfigurationDataBase%

@Echo 4) Import .msi (Before this Undeploy the existed policies, the /Overwrite doesn't help! Or Stop the Application.)
BTSTask ImportApp /Package:"%CurrentBuildPath%\Policies.msi" /ApplicationName:%ApplicationName% /Overwrite /Server:%BizTalkDataBasesServer% /Database:%BizTalkConfigurationDataBase%

@Echo 5) Deploy policies one by one :((

@set PolicyName=VesselID
@set PolicyVersion=1.27
BTSTask AddResource /ApplicationName:%ApplicationName% /Type:System.BizTalk:Rules /Overwrite /Name:%PolicyName% /Version:%PolicyVersion% /Server:%BizTalkDataBasesServer% /Database:%BizTalkConfigurationDataBase%

@set PolicyName=CompanyID
@set PolicyVersion=1.0
BTSTask AddResource /ApplicationName:%ApplicationName% /Type:System.BizTalk:Rules /Overwrite /Name:%PolicyName% /Version:%PolicyVersion% /Server:%BizTalkDataBasesServer% /Database:%BizTalkConfigurationDataBase%

@set PolicyName=PortOfUnloadingID
@set PolicyVersion=1.0
BTSTask AddResource /ApplicationName:%ApplicationName% /Type:System.BizTalk:Rules /Overwrite /Name:%PolicyName% /Version:%PolicyVersion% /Server:%BizTalkDataBasesServer% /Database:%BizTalkConfigurationDataBase%

@set PolicyName=DocumentValidator
@set PolicyVersion=1.1
BTSTask AddResource /ApplicationName:%ApplicationName% /Type:System.BizTalk:Rules /Overwrite /Name:%PolicyName% /Version:%PolicyVersion% /Server:%BizTalkDataBasesServer% /Database:%BizTalkConfigurationDataBase%

@Time /T

@rem call RestartBTS.bat

pause

@rem ----------- Settings.bat ---------------------------------------------
@set MyShortSolutionName=MyShortSolutionName
@Set SolutionName=MyCompany.Domain.%MyShortSolutionName%
@set SolutionDir=C:\Solutions\%MyShortSolutionName%
@SET CurrentBuildPath=%SolutionDir%\Builds\Current
@SET BizTalkServer=BTS-06V
@SET BizTalkDataBasesServer=BTS-06V
@SET BizTalkConfigurationDataBase=BizTalkMgmtDb
@Set ApplicationName=%MyShortSolutionName%.MyApp

BizTalk: Anti-Patterns: Business Rule Engine

Leonid Ganeline — Thu, 28 Dec 2006 03:56:00 GMT

Let's see one of the projects I've made with the Business Rule Engine from the Microsoft BizTalk tool set.
There are the EDI data in"very flexible format". The source partners fill up the fields of these documents by different applications and peoples. As a result the data are usually spread through the several fields if they filling application has a limit to the field size.

For example:
Good data:

N1*CN*YOUNG BAY INTERNATIONAL INC
N3*2655 ST-JACQUES#101*MONTREAL QC H3J 1H8 CANADA
N3*TEL:514-8121887 514-9313157

Bad data:

N1*CN*AFS -VISA INTERNATIONAL (HK) LTD    SUITE NO.1 29/F SKYLINE T
N3*OWER      39 WANG KWONG ROAD, KOWLO
N3*ON BAY,   KOWLOON, HONG KONG

The data are in the wrong fields; the data are spread through the several fields; etc.

I need to map the data to the into the local database. For example:
we've got the source data: "YOUNG BAY INTERNATIONAL INC", which can be mapped to several "YOUNG BAY"-like companies, depends of other source data. In our data base we have:
"YOUNG BAY INC"
"YOUNG BAY INT'L CO"
"YOUNG-BAY INTERNATIONAL "
...
plus the variation in the other source data as City, State, CountryCode, PostalCode, Address etc.

This project is the good candidate to use BRE. The data in the input set has different relations (rules). Those rules tend to be constantly improved and modified. It is the ideal candidate to use BRE, isn't it?
We can process the data by BRE. We can easily develop and modify the sets of rules.

The project was implemented in two stages.
First was the "Unification of the data". It unified the punctuation, "Incorp" --> ""Inc", "Logictic" -> "Logistics", "HONGKONG" -> "HK", "U S A" -> "US", "H3J 1H8" -> "H3J1H8", etc. etc. etc.
The second stage was the "Company resolving". It took the unified data from the input documents and looked through the corporate data base to map the input data to the corporate data. The example is above.

This part was implemented with help of the BRE .
The Rule Sets were implemented. There were about forty rules.

And from the start there were the big issues with performance.
Why? The most of the rules did not have the conditions with the simple boolean comparisons like "<Name_Compny> == "value1" but the resource consuming query to the data base.
In the BRE the MOST of the rule conditions are tested (invoked) at the start. (For explanation there is a great article by Charles Young http://geekswithblogs.net/cyoung/articles/79500.aspx) In my case near ALL forty data base queries hit the performance the system.
I did not need in all queries at the start. Then I separated the rules to stages. If the first set of rules did not get the result, the second set comes to the work, etc.
Sounds good? Let see.
One of the BRE technology advantage is a Rete networks (a Rete algorithm) (see http://en.wikipedia.org/wiki/Rete_algorithm). It gives us the capability to efficiently process the big or huge sets of data. The BRE takes care about the order of the tests in the rule set conditions. Moreover the BRE cached the fields of an XML document or the columns of a database table and when these data are accessed for the second time or later within the policy, the values are usually retrieved from the cache. However, when the members of a .NET object are accessed for the second time or later, the values are retrieved from the .NET object, and not from the cache. That means if I invoke the same "test" method from several rules, BRE do not cache the result of this method but each time re invoke the method. It was unfortunately my case.
User has few techniques to manage the order and I used the simplest. I added to each test (or stage) a flag which goes up after this test is ended. The conditions checked the flags. The down flags in the conditions prevented the heavy part of the condition from the start. Now the conditions were started in managed order.

Looks weird.

First of all, the BRE is used to separate the business logic to the independent layer to make it easy (easy to creating, modification, maintaining).
Did I gather the business logic in the rules? No. The main part of rules was remained in the database queries and the classes which wrapped those queries. The rules did only manage the invoking of these queries. If these layers were the same size then the layering give me improvement. The most of the logic was out of BRE Rules layer and easily could be implemented by .NET code. Maintaining the BRE infrastructure for such easy work as invoking the methods in my case was not a good choice.

Second. The BRE takes care about the order of the tests in the rule set conditions. It SHOULD take care of it, otherwise we throw away of this advantage of the Business Rule Engine. If we manage the rule set condition order, we stopped at the same point where we started. I mean now I used BRE, it was cool, but all the rule order processing was mine responsibility. And the most of my issues was to avoid the BRE most amazing feature.

If the BRE was managed the rule set condition order, the performance was bad.
If I managed the rule set condition order, why do I need the BRE?

After several modifications in the rules I've decides to redesign the project. I've created the simple class to manage the rules and got rid of the BRE artifacts.
Ufff... It was cool but in my case it was something different.
Now all staff is pretty simple and manageable.
You can tell me "Now you cannot use the BRE Composer as a User interface to modifying and improving the business layer logic." That's true. Now I cannot delegate the "rule-modification" job to the third party. If this is a requirement then the BRE and the Composer would be a good choice.

Conclusions ("Donts"):

1. If your rules have the resource-consuming conditions (predicates) be aware of non-efficient performance.
2. If your rules have the well-defined processing order, seems the BRE is not your choice. If you don't care about the rule processing order, the BRE could be useful.
3. If your rules have the values in the conditions retrieved from the .NET object be aware of non-efficient performance.
4. Rules should cover the biggest part of the business logic. If the rules have a lot of calls to the .NET classes think of redesign this project.

Be aware of the BRE:
It is the tool for processing the specific data set with specific conditions. Not all data and not all business logic can be implemented efficiently with BRE.
Maybe it is not what exactly we can estimate from the product with such name. Now it is hardly a tool for the Business Analyst. It is the tool for software developers. And this is great tool!
I'm sure the next version would be much better from a developers point of view.

Notes:
* The other simple method to stage the rules is to separate the rules to different policies and chain these policies. This approach did not work well for my case because I mostly had to manage the order of the rules, not the stages, what leads to pack one rule to one policy. Each policy creates the persistence point in the orchestration. Maybe it would not hit the performance too hard in my case, I didn't try this.

* We can force the caching of the .NET objects but only programmatic. The Business Rule Composer tool does not expose this possibility.

BizTalk: Business Rule Engine: Don't use vocabulary in development

Leonid Ganeline — Mon, 16 Oct 2006 19:09:00 GMT

BizTalk: Business Rule Engine:
Don't use vocabulary in development!

Why?

Because it gives you big headache and so little instead of.

Our typical work with the facts in the rules is compounded from the steps:

* Changing/Add fact in/to a rule (if fact is in the one of the tabs of Fact Explorer): Without vocabulary we change/add fact in a rule by drag-and-drop it from the Fact Explorer. With vocabulary we drag-and-drop a definition from the Fact Explorer. No difference.

* Changing/Add fact in/to a rule (if this fact is not in vocabulary):
Without vocabulary we work as before. With vocabulary we create a new version of our vocabulary (we can not change the old version because it is published (we have to publish the vocabulary otherwise we can not use its facts in the rules)).

* Cut the old version of the vocabulary:
In the rules we change all facts from the old version to the facts from the new version. If we have some facts in vocabulary with reference to it from some rule we can not delete the old version of vocabulary with this fact. (I know the free utility for this purpose but don't recommend it because I had the unpredictable behaviour of the rules after such automatic updates of the facts.)
In the real work it looks like: OK. This rule is near finish. I create a rule set but it use several versions of the one vocabulary and I don't like it. I want to cut the old versions. (I don't like to deploy many versions of one vocabulary. It creates absolutely mess.) I have to manualy upgrade the facts, change old version to a new one, then delete old versions of the vocabulary. But... the vocabulary is "near" finish. Oops, I need change one more fact inside (and create one more version). It's always "near"...
When I work with the rule I can cange and it, change it without creating a new rule version. But with vocabulary it is completely prohibited. I can use vocabulary and test it only after publishing ("freeze and use").
OK. I'm trying to separate one vocabulary to two. One is the "Stable", the second is "Mutable". It can make my life easy. Wow, how interesting. I was thinking the vocabulary helps me to keep in mind less entities, but now I have to create an additional infrastucture...
BTW I don't understand why we can't freeze the vocabulary with policy together.

One more time.
A new fact: Without vocabulary I drag-drop new fact to the rule.
With vocabulary I have to:
1. create a new version of the vocabulary.
2. add a new fact to it. And it is NOT short way! I do not describe here the odds of this step.
3. Save and Publish (and, maybe, Deploy) new version.
4. drag-drop new fact from vocabulary.

Additional work:
1. creating a definition in the vocabulary.
2. changing the old definition in the rules on the new definition from the last version of vocabulary.
3. managing versions of the vocabulary.
4. in deployment: creating, deploying and managing the vocabulary.

Pro for Vocabularies:
* short names of facts in the rules. But in the rules we do not see "very long names" (not like "xpath-names" in maps: "/*[local-name()='EDI' and namespace-uri()='http://VPA.EDI.Schemas.EDICanonical']/*[local-name()='VesselID' and namespace-uri()='']"), we see readable names as "VPA.EDI.Schemas.EDICanonical:EDI/VesselID". It is very "small pro".
* readable names and formats for the facts. It is a good Pro for the end users.

Conclusion: I can see only one case when a vocabulary worth all this additional headache: If we have very stable rule set in the production and we can give it to the end power users (BTW never seen this mythic user).
In the real development never use vocabulary!

Some Ideas

I won't use the vocabularies now.

Anyway the idea of the vocabulary is wonderful.

In what conditions I'd rather like to use them?

Lets talk.

I'd like the vocabulary be in sync with a policy. That means the changes in a vocabulary should not mean the changes in the policy. Let changes in the policy occur by default.

But the vocabulary is the separate from the policies entity. It can be used by several policies simultaneously.

Lets think about it. One of the solutions is this:

By default a vocabulary is the part of the policy (BTW In the BizTalk it is the part of the policy. When we deploy the policy, we deploy the vocabularies in use with this policy.). In the BR Composer it should be the branch under the Policy node. All operation under Policy: Save, Publish, Deploy, Undeploy should work under (Policy+Vocabulary)ver.xx entity as whole. In this case I don't care about versions of the definitions in the current policy they are always in sync.

Lets save the current separate vocabularies, but call them "shared vocabulary". If somebody wants to use the shared vocabulary, why don't.

--
Regards,

--
Leonid Ganeline
BizTalk Solution Developer