Geeks With Blogs

News View Michael Stephenson's profile on BizTalk Blog Doc View Michael Stephenson's profile on LinkedIn
Michael Stephenson keeping your feet on premise while your heads in the cloud

Article Source: http://geekswithblogs.net/michaelstephenson

I've recently been looking at a proof of concept of implementing the resequencer pattern in BizTalk 2009. When I've implemented this pattern in the past it usually required a lot of custom development to create a database to support the resequencing process. If you would like to see more on the theory of the resequencer pattern then refer to the following link: http://www.eaipatterns.com/Resequencer.html.

I've seen a few .net and BizTalk examples which discuss doing the resequencing in memory, but for my case I felt this would be a limiting factor as it would mean I could only use 1 machine to implement this on and there could be scalability issues. The aims of my proof of concept were as follows:

  • To create a demonstration of the resequencer pattern that would be scalable to handle lots of messages in BizTalk
  • Be able to extend this relatively easily to handle resequencing messages from multiple sources. For example if you had many partners pushing in messages and you only had to resequence messages from each partner separately
  • Minimise custom development and maintenance requirements to support this. I didn't want to have any custom databases in my solution.
  • Try to make the resequencing pattern implementation as reusable as possible

Before I got into the details of this I was already familiar with the Frends product which I have discussed how it could be used to compliment BizTalk solutions in previous articles. My thought was that I might be able to use the configuration store part of Frends to help me to manage the sequencing in my POC. The rest of this article will discuss how my POC works.

My POC Environment

The environment on which I have build my POC is a Windows 2008 virtual PC running the CTP of BizTalk 2009 and SQL Server 2008. I have also installed the trial version of Frends.

Configuring Frends

I would need to configure a "Connection Point" in Frends. A connection point is basically an instance of some configuration data. You can set these up through the Frends Management Console where you will provide a schema which will define your configuration data, and then in Frends you specify values for the schema elements. To do this I took the following actions:

  1. Create Scheduling Configuration Schema

In my Visual Studio solution I created the below schema. In this schema I have defined an element to hold the sequence number of the last message processed, and an element to hold the time the last message was processed.

 

  1. Create Connection Point

In the Frends Management Console I would choose to add a new Connection Point like in the below picture.

 

On the add new Connection Point screen you need to supply the name of the connection point and choose a schema which will represent the configuration data. I have chosen the schema I created above and you can see in the below diagram that it will create a designer to allow you to supply values for the configuration. To begin with I will leave the time element blank, but I will supply the value zero to be the last processed message as it will indicate the start of the sequence. One interesting thing about the connection point is that you can have multiple connection points with the same name if you supply a different Party Name for each. This lets you correlate a specific connection point instance to a party you may be dealing with.

In the picture below you can see my configured Connection Point.

 

Setting up my BizTalk Solution

In setting up my BizTalk solution I have two projects. The first project will contain my BizTalk artefacts and the second will contain some utility components. These are shown in the below picture.

The key points of the solution are described in the below table:

Artefact

Description

InputSchema.xsd

This is the message schema that will be input into the process

SequencedOrchestration.odx

This is an orchestration which will start an instance for each message added, and will not continue until it is the next message to process.

SequencingFacade.cs

This class will implement a façade pattern to the Frends system

 

The input message will look like the below picture. It simply is a message containing the number of the message.

For each message that is received an instance of the SequencedOrchestration will start. The below picture shows the orchestration and I have highlighted some of the key points to discuss:

  1. The loop will continue until the variable which indicates this instance is the next in sequence is set to true
  2. This expression shape will call the SequencingFacade to work out if this instance is next
  3. If this instance is not next then the instance will delay for a while
  4. When this instance is next the loop exists and I will create the message to send out. I also set the File Adapter context property do I can out put a file name which is the same as the sequence number
  5. This expression shape will call the SequencingFacade to update the last processed sequence number

 

The Sequencing Façade

In the SequencingFacade class the first method to discuss is the AmINext method. This method will call Frends to obtain the connection point information. Frends makes the Connection Point subsystem available as a web service using WCF. Once I have called the web service I can get the configuration data from the response as a string of XML. Because I'm using BizTalk 2009 and can compile the .net 3.5 features in my solution I have chosen to use Linq to XML to parse this XML to get the sequencing details.

I can then implement some simple logic to work out if the instance is the next in sequence which you can see in the below picture.

The next method in the SequencingFacade is the UpdateSequence method. In the below picture you can see I get the sequence information from Frends and then use Linq to XML to update the xml to indicate a message has been processed. I finally call the Frends web service to update the connection point.

 

Walk Through

Now that we have reviewed the highlights of this implementation I will walk through a sample of how it works.

  1. We are currently at the start of the process and I have configured Frends that the last message processed was zero. The output folder contains no messages and there are no instances running in BizTalk.
  2. I add message with sequence 1 into the input folder. This message is collected and processed. Message 1 ends up in the output folder, and the Frends connection point is updated as in the picture below

  1. I now add message 3 to the folder. You will that this instance will be running in BizTalk and will keep checking Frends to see if its next, and will keep using the Delay shape between checks.

  1. I will now add message 4 to the folder. You will see this will also continue waiting like message 3.

  1. I finally will add message 2 to the input folder. You will see that it will process first and complete, then message 3 and 4 will follow it out in order. You will be able to see all messages in the output folder and finally Frends will know that the last message processed was 4

  1. I also has debugview running during the walkthrough and you can see from some of the trace messages I recorded that the messages were processed as expected

     

     

     

Summary

My proof of concept proved to be quite successful. I was able to setup this demonstration very quickly and it did everything I required. I think I have been able to show an implementation for the resequencer pattern which will be quite scalable and able to handle a lot of messages by taking advantage of the BizTalk scalability model and its ability to process lots of orchestrations at the same time. Some of my other thoughts about this pattern which may help move this to a real world implementation are as follows:

  • I would be looking to use the party name bit of Frends to help me handle messages coming from different sources
  • I would probably look to use something like NCache or Velocity to do a distributed caching of the responses from Frends if I was going to have a high load scenario. I need to have the permenant storage which Frends provides but by caching the sequence number and updating the cache when a process is complete I would not need to call the Frends web services very often. Also using one of the distributed caching solution would mean I can synchronise the cache across multiple machines within my BizTalk group
  • I have coded the interaction with Frends in C#, but it could equally be done in an orchestration by calling the services with a send port etc.  The Frends service would normally be running on your BizTalk machine anyway so I guess you could choose how you did this
  • I could probably implement the whole loop in a seperate orchestration where you would pass in the name of the connection point (and optional party) and your sequence number to this orchestration which would then interact with Frends.  This means you could potentially implement resequencing of any of your orchestrations by just calling this other orchestration (note this will mean you have more running instances)

If you would like to have a look at the sample code then it is available on box.net at the below link. (Note some companies block box.net so if you are unable to get the sample then let me know)

http://www.box.net/shared/ircluappxe

 

 

Posted on Thursday, January 1, 2009 11:11 PM BizTalk | Back to top


Comments on this post: Implementing the Resequencer Pattern with BizTalk 2009 and Frends

# re: Implementing the Resequencer Pattern with BizTalk 2009 and Frends
Requesting Gravatar...
This is a handy sample indeed, but requires FRENDS which is a 3rd party component that comes at a cost. What else does FRENDS do in this environment other than just maintaining the sequence information?
Left by Ryan CrawCour on May 02, 2010 5:58 PM

# re: Implementing the Resequencer Pattern with BizTalk 2009 and Frends
Requesting Gravatar...
Michael,
Have you ever implemented this in the wild? I proposed using this pattern for resequencing in one of my projects but one of my colleagues questioned whwther this pattern really does ensure ordered delivery on the send port. They pointed me in the direction of this discussion of resequencing (http://social.msdn.microsoft.com/Forums/en-US/biztalkgeneral/thread/b4efa786-8e60-4e0c-b603-e70e9593287a ) which does not mention your pattern. I have also just re-read the article on FIFO processing in the 1st issue of the BizTalk Hotrod magazine and that suggests your pattern of multiple orchestration instances will not ensure order. I would be really interested to hear your comments. Regards Mark
Left by Mark Brimble on Aug 10, 2010 4:15 AM

# re: Implementing the Resequencer Pattern with BizTalk 2009 and Frends
Requesting Gravatar...
Thanks Mike
I have gone through a similar thought process to you. I am halfway through writing an implementation using a custom database as you described to prove to my colleague that it does work. I would very interested to see your post using AppFabric.
Left by Mark Brimble on Aug 11, 2010 9:54 PM

# re: Implementing the Resequencer Pattern with BizTalk 2009 and Frends
Requesting Gravatar...
Ludwig.Nach the summer holidays start t men's clothing,he roofing and carpentry works.

The tender submission period for the

Installation work ends o weddings,n Monday, August 22. Applications from firms can until then filed

be.
Left by linming0303 on Nov 08, 2017 6:44 PM

Your comment:
 (will show your gravatar)


Copyright © Michael Stephenson | Powered by: GeeksWithBlogs.net