The first BizTalk project I got involved with involved designing a message aggregator using a sequential convoy, and being new to BizTalk at the time, it was a real struggle to build even a basic prototype. After a few failed attempts, I have come up with an implementation that seems to work quite well, and could be used as a reference for creating more complex message aggregators.
The implementation is based on the Aggregator pattern form “Enterprise Integration Patterns”, more information here.
Problem
An order processing system sends individual order messages to the message hub. In order to reduce system load, the suppliers have requested that order lists containing multiple orders should be sent to them.
The individual order messages should be combined into an order list and dispatched to the suppliers. Each supplier should receive no more than one order list per minute.
Designing the Aggregator
When designing a message aggregator, three factors need to be considered:
Correlation – How are the incoming messages going to be grouped together
Completeness Condition – How do we decide when the aggregated message is complete
Aggregation Algorithm – How should the inbound messages be combined to form the result message
For our example the first two can be answered easily, the third requires more thought:
How are the incoming messages going to be grouped together?
The SupplierID in the order message will be used to group the order messages for each supplier.
How do we decide when the aggregated message is complete?
We will aggregate the messages for one minute, then send the order list massage.
How should the inbound messages be combined to form the result message?
As BizTalk uses XML messages internally, we will need to play around with XML document manipulation.
BizTalk Implementation
Ok, how do we get this to work in an orchestration?
BizTalk provides us with a few features that are useful for this scenario:
- An orchestration can be used for the aggregation process, and to hold the state of the order list messages during this process
- Correlation can be used to identify the inbound order messages for each supplier.
- XLANG/s can be used within an orchestration to manipulate the XML messages
- Various orchestration shapes can be used to control the flow and decide when the completeness condition is reached.
Orchestration Description
The orchestration diagram is shown here:
The main shapes used are as follows:
|
RcvOrder |
Activates the orchestration and initialises the SupplierID correlation set |
|
SetOrchVariables |
Stores the supplierID, sets the time out time and flag, and creates an XmlDocument for storing the OrderList message |
|
ReceiveOrdersLoop |
Loops until the timeout flag is true |
|
MapOrderToIntOrder |
Maps the Order message to an InternalOrder message |
|
AddOrderScope |
The XmlNode object is not serialisable, so we need to perform the aggregation in an atomic scope. |
|
AggrigateOrderMsg |
Performs the message aggregation, adding the Order to the XmlDocument |
|
ListenForNextOrder |
Wait until the next order arrives or the timeout time is reached |
|
RcvNextOrder |
Follows the SupplierID correlation set. |
|
DelayUntullTimeout |
If the timeout occurs, this side of the ListenForNextOrder will execute. |
|
SetTimeoutFlag |
Set the timeout flag for exiting the loop. |
|
SetOrderListMessage |
Convert the XmlDocument into an OrderList message |
|
SndOrderList |
Sends the OrderList message. |
Solution Files
The Aggregator project files are located here.
Build
To install project with the default file adapter locations, copy the files into the folder:
C:\BizTalkPatterns\Aggregator
Build and deploy in the normal way, or use the NAnt Build Script.
Test
Copy the SupplierXOrder.xml files in the IN directory. The orchestration should aggregate the files and produce one OrderList file per minute for each supplier. Check the running instances in HAT; you should see one for each supplier.
Implementing Aggregators
Here are a couple of tips for building your own aggregators:
- Keep any heavy processing or calls to other systems outside the aggregator. As the aggregator is a singleton instance, and messages are processed sequentially, a 10 sec call to a web service for each message will limit your throughput to 6 messages a minute, also, any errors or exceptions that occur with the call could suspend the whole aggregator. Using a separate orchestration to make this call for each message means BizTalk can process the order messages in parallel giving much faster throughput, with any call exceptions only affecting a single order.
-
You can use a more complex aggregation algorithm, it this scenario, it might be an idea to group multiple orders for the same item together in one order. To keep life simple, I have implemented the algorithm in an expression shape; using a .net class to do this would allow more complex algorithms.
Feedback
Feel free to comment on the design; if there are any good points on improving it, I will update the project.