Geeks With Blogs

News
CloudCasts Blog Webcasts in the Cloud

The first BizTalk project I got involved with involved designing a message aggregator using a sequential convoy, and being new to BizTalk at the time, it was a real struggle to build even a basic prototype. After a few failed attempts, I have come up with an implementation that seems to work quite well, and could be used as a reference for creating more complex message aggregators.

The implementation is based on the Aggregator pattern form “Enterprise Integration Patterns”, more information here.

 

Problem

An order processing system sends individual order messages to the message hub. In order to reduce system load, the suppliers have requested that order lists containing multiple orders should be sent to them.

The individual order messages should be combined into an order list and dispatched to the suppliers. Each supplier should receive no more than one order list per minute.

 

Designing the Aggregator

When designing a message aggregator, three factors need to be considered:

 

Correlation – How are the incoming messages going to be grouped together

 

Completeness Condition – How do we decide when the aggregated message is complete

 

Aggregation Algorithm – How should the inbound messages be combined to form the result message

 

For our example the first two can be answered easily, the third requires more thought:

 

How are the incoming messages going to be grouped together?

The SupplierID in the order message will be used to group the order messages for each supplier.

 

How do we decide when the aggregated message is complete?

We will aggregate the messages for one minute, then send the order list massage.

 

How should the inbound messages be combined to form the result message?

As BizTalk uses XML messages internally, we will need to play around with XML document manipulation.

 

 

BizTalk Implementation

Ok, how do we get this to work in an orchestration?

 

BizTalk provides us with a few features that are useful for this scenario:

 

  • An orchestration can be used for the aggregation process, and to hold the state of the order list messages during this process
  • Correlation can be used to identify the inbound order messages for each supplier.
  • XLANG/s can be used within an orchestration to manipulate the XML messages
  • Various orchestration shapes can be used to control the flow and decide when the completeness condition is reached.

 

 

Orchestration Description

 

The orchestration diagram is shown here:

 

 

 

The main shapes used are as follows:

 

RcvOrder

Activates the orchestration and initialises the SupplierID correlation set

SetOrchVariables

Stores the supplierID, sets the time out time and flag, and creates an XmlDocument for storing the OrderList message

ReceiveOrdersLoop

Loops until the timeout flag is true

MapOrderToIntOrder

Maps the Order message to an InternalOrder message

AddOrderScope

The XmlNode object is not serialisable, so we need to perform the aggregation in an atomic scope.

AggrigateOrderMsg

Performs the message aggregation, adding the Order to the XmlDocument

ListenForNextOrder

Wait until the next order arrives or the timeout time is reached

RcvNextOrder

Follows the SupplierID correlation set.

DelayUntullTimeout

If the timeout occurs, this side of the ListenForNextOrder will execute.

SetTimeoutFlag

Set the timeout flag for exiting the loop.

SetOrderListMessage

Convert the XmlDocument into an OrderList message

SndOrderList

Sends the OrderList message.

 

 

Solution Files

The Aggregator project files are located here

 

Build

To install project with the default file adapter locations, copy the files into the folder:

 

C:\BizTalkPatterns\Aggregator

 

Build and deploy in the normal way, or use the NAnt Build Script.

 

Test

Copy the SupplierXOrder.xml files in the IN directory. The orchestration should aggregate the files and produce one OrderList file per minute for each supplier. Check the running instances in HAT; you should see one for each supplier.

 

 

 

Implementing Aggregators

 

Here are a couple of tips for building your own aggregators:

 

  • Keep any heavy processing or calls to other systems outside the aggregator. As the aggregator is a singleton instance, and messages are processed sequentially, a 10 sec call to a web service for each message will limit your throughput to 6 messages a minute, also, any errors or exceptions that occur with the call could suspend the whole aggregator. Using a separate orchestration to make this call for each message means BizTalk can process the order messages in parallel giving much faster throughput, with any call exceptions only affecting a single order.
  • You can use a more complex aggregation algorithm, it this scenario, it might be an idea to group multiple orders for the same item together in one order. To keep life simple, I have implemented the algorithm in an expression shape; using a .net class to do this would allow more complex algorithms.

 

 

Feedback

Feel free to comment on the design; if there are any good points on improving it, I will update the project.

 

 

Posted on Monday, August 16, 2004 8:12 PM | Back to top


Comments on this post: Sequential Convoy Aggregator Pattern in BizTalk Server 2004

# re: Sequential Convoy Aggregator Pattern in BizTalk Server 2004
Requesting Gravatar...
Great article, and very useful inhelping me grasp the power of the new orchestrations over the old.

I'm simply trying to ensure that each message received is delivered in chronicalogical order.

Can this be done?

thanks again,
Conor
Left by Conor Maguire on Oct 14, 2004 10:40 AM

# re: Sequential Convoy Aggregator Pattern in BizTalk Server 2004
Requesting Gravatar...
Hi,

If you are using an adapter like the file, or FTP adapter, there is no guarentee that the messages will enter the aggregator in the correct order.
If you want to ensure the messages are aggregated in chronological order, you will have to implement this in the aggregation algorithm. To do this, you could use a sorted collection in .net, and use a timestamp to sort the messages by.

Regards,

Alan
Left by Alan on Oct 14, 2004 1:28 PM

# re: Sequential Convoy Aggregator Pattern in BizTalk Server 2004
Requesting Gravatar...
Hi,

What happens if a message comes after the timeout but before the current orchestration completes? does this start a new instance?

Thanks
Vijay
Left by Vijay on Jan 04, 2005 7:29 PM

# re: Sequential Convoy Aggregator Pattern in BizTalk Server 2004
Requesting Gravatar...
Hi Vijay,

Yes, it will start a new instance of the orchestration.

/Alan
Left by Alan Smith on Jan 05, 2005 2:11 PM

# re: Sequential Convoy Aggregator Pattern in BizTalk Server 2004
Requesting Gravatar...
Hi Alan and Vijay,

Regarding the last comment, I'm not absolutely sure that it will actually start a new instance of the orchestration.

I just implemented a sequential convoy and it appears that sometimes, when a message arrives after the timeout occur and that the current orchestration is not terminated yet, that message may subscribe to the orchestration anyway so it completes "with discarded messages" (that's the status of those orchestration instances in the HAT). This is what some people call "zombies".

My orchestration currently process about 1000 messages per day and there is about 1 or 2 messages each day that still unprocessed.

Any thoughts on that?

Thanks,
Gely
Left by Gely on Mar 17, 2005 2:06 PM

# re: Sequential Convoy Aggregator Pattern in BizTalk Server 2004
Requesting Gravatar...
Hi Gely,

What if to start a new Orchestration instance each time when this aggregator orchestration ends, either by itself or by some super (caller) orchestration. This might ensure that no message are discarded/skipped.

Thanks,
Farrukh
Left by Farrukh on Mar 31, 2005 11:50 AM

# re: Sequential Convoy Aggregator Pattern in BizTalk Server 2004
Requesting Gravatar...
I set up a similar process, but when compiling it I get 'Microsoft.XLANGs.RuntimeTypes.XmlDocumentSerializationProxy' does not contain a definition for 'CreateNode'

Why would I be getting this?

Thanks,
Eric
Left by Eric Stott on Oct 08, 2005 6:06 AM

# re: Sequential Convoy Aggregator Pattern in BizTalk Server 2004
Requesting Gravatar...
Have you got any solution for the zombies issues u have mentioned ? Please update on that.

Article was excellent and clear.
Left by vijay on Mar 11, 2008 1:03 PM

# re: Sequential Convoy Aggregator Pattern in BizTalk Server 2004
Requesting Gravatar...
Hi Vijay,

Yes, it will start a new instance of the orchestration.

/oyun indir
Left by oyun indir on Jul 06, 2008 6:31 PM

# re: Sequential Convoy Aggregator Pattern in BizTalk Server 2004
Requesting Gravatar...
in regard to Vijay's question about whether or not a new instance would be created; i am tending to think not. you will end up the potential for zombies.
zombies occur when the orchestration has moved past the delay shape but before it completes. if the orchestration completes and THEN a new message arrives a new instance will be started.
Left by Ryan CrawCour on Nov 03, 2009 9:59 AM

# re: Sequential Convoy Aggregator Pattern in BizTalk Server 2004
Requesting Gravatar...
Hi,
Is there a possibility that I can get the code for this article. Tried the link but looks like the code has been removed.

Cheers,
RC
Left by RC on Mar 14, 2010 6:26 AM

Your comment:
 (will show your gravatar)


Copyright © Alan Smith | Powered by: GeeksWithBlogs.net