Debatching Into Smaller Batches Inside An Orchestration in BizTalk 2004

<< WMI Suspended Instance Application for BizTalk 2004 | Home

Debatching Into Smaller Batches Inside An Orchestration in BizTalk 2004

It is a common request. How can I debatch a message into smaller batches rather than single messages inside BizTalk 2004?

I have tried many times to get this accomplished natively inside the pipeline. One would think it would be as easy as setting the group property to 10 if you wanted a batch size of 10, but that does not seem to work. The only way I have been able to break up a message into smaller batches has been inside custom code.

I wrote a .net component that can be called using an atomic scope shape from inside an Orchestration. This component will return smaller batches of any size you specify inside the Orchestration and wrap the smaller batches inside a specified root node. This .net component uses the XmlTextReader to stream the message along and debatch it.

This .net component is called StreamHelper. It illustrates the following:

Using XmlTextReader
Passing an Xlang Message into a .net component
Returning an Xlang Message from a .net component
Setting context properties on a message returned from a .net component

Download: Debatch Streaming Sample

Setup: Download and extract the solution. Build and GAC the StreamHelper assembly then Build and Deploy the DebatchProperties and DebatchOrch assemblies. I have included a basic sample file to test with. For more detailed information on the sample see the comments throughout the solution.

A better way to do it?

I’m sure there is. I just put this together in a short amount of time. I would guess the most efficient place for this type of work would be inside a custom pipeline component. But, working with custom pipelines can be difficult.

How does this debatching approach perform compared to the others?

Performance was almost identical to that of the Xml Node List inside an atomic scope. In this sample, I used the XmlTextReader and passed Xlang messages in and out of the component to try to increase performance. That did not seem to accomplish anything. As expected, as the total number of messages inside the atomic scope increased the performance decreased.

Type	XML Size (MB)	# Nodes	Batch Size	Time (Sec)	Msg/Sec	Msg Size (kb)
.net Stream	1.6	500	1	5	100.0	3.0
.net Stream	3.6	1100	1	18	61.1	3.0
.net Stream	7.2	2200	1	44	50.0	3.0
.net Stream	18.1	5500	1	224	24.6	3.0
.net Stream	128.6	38500	1	--	FAIL	--
.net Stream	1.6	500	10	1	500.0	3.0
.net Stream	3.6	1100	10	3	366.7	3.0
.net Stream	7.2	2200	10	9	244.4	3.0
.net Stream	18.1	5500	10	15	366.7	3.0
.net Stream	128.6	38500	10	159	242.1	3.0

For more debatching information please see my past post:
Debatching Options and Performance Considerations in BizTalk 2004

Print | posted on Monday, March 21, 2005 11:38 PM |

Feedback

# re: Debatching Into Smaller Batches Inside An Orchestration in BizTalk 2004

Hi Stephen,

Interesting post. On one of the entries in the table it says FAIL. What happened there?

3/22/2005 12:08 PM | McGeeky

# re: Debatching Into Smaller Batches Inside An Orchestration in BizTalk 2004

Oh, I guess I didn’t talk about that. I was running a very large file, 128 MB, and the system never produced any output after 45 minutes. The CPU went to zero but nothing was happening. I had to shut down the host and terminate the still Active Orchestration. I think the total number of messages inside the atomic scope was too much for my system.

Note that the same file with a large batch size worked just fine.

Thanks for pointing this out.

Stephen W. Thomas

3/22/2005 12:49 PM | Stephen W. Thomas

# BizTalk Flat File Debatching Design Pattern: Decoupling the Debatching from the Processing, Part I

Debatching flat files is pretty typical BizTalk fare. However, as I can attest, getting BizTalk to de-batch a flat file so that it can deal with each flat file row as a separate entity while not terminating the entire batch if a row is invalid is less than straightforward.

9/28/2005 1:06 PM | Michael G. Emmons' C#, .NET and

# re: Debatching Into Smaller Batches Inside An Orchestration in BizTalk 2004

The example was very good and is helping alot.

I got it to work but with a little problem. It only accepts and processes an
XML file that is structured correctly. i.e. when opening it in notepad, it
looks like this:

<testfile>
<Body>
<Data>Anything or any other nodes</Data>
</Body>
<Body>
<Data>Anything or any other nodes</Data>
</Body>
</testfile>

However, the XML files I would like to process (initially recieved from
BizTalk 2004) looks something like this:

<testfile><Body><Data>Anything or any other
nodes</Data></Body><Body><Data>Anything or any other
nodes</Data></Body></testfile>

This failed for some reason, and it only produces a single record ignoring
the rest of Body records in the file above!!

Any idea why that is and where I need to modify to get it to accept the
above format as this is the normal XML format that BizTalk 2004 produces!

Kam

1/19/2006 2:30 AM | Kam

# re: Debatching Into Smaller Batches Inside An Orchestration in BizTalk 2004

I noticed what as well when I built the sample but I didn’t think much of it. I have seen that before on a client project as well. I don’t really understand it and I was not able to correct it. The pipeline didn’t seem to make any difference.

Long and short of it is I do not think I can help much on this one but if you figure it out please let me know.

Stephen W. Thomas

1/19/2006 8:50 AM | Stephen W. Thomas

# re: Debatching Into Smaller Batches Inside An Orchestration in BizTalk 2004

Just to let you know that I got it to work by changing the Grid property of the map (Indent = Yes).

Changing the above property to yes created an indented XML file which allowed the StreamHelper code to read all nodes in the file.

The only problem with that is the size of the file has increased from 82MB to 94MB.

Anyhow, I can live with this for now but it would be better to find a way to read a non-indented XML file using the XmlTextReader in StreamHelper class.

For info, debatching the file without applying the map took between 15-20 minutes on my Virtual PC. I'll be testing it on an actual machine and let you know of the result.

Thank you.

Kam

1/20/2006 3:34 AM | Kam

# re: Debatching Into Smaller Batches Inside An Orchestration in BizTalk 2004

Tried to add a delay shape at the end of the loop to send the files in an interval of 2 mintues each but failed as it is an atomic scope, I believe.

Is there any other way of sending the de-batched files having a delay between each one?

11/7/2006 8:45 AM | Kam

# re: Debatching Into Smaller Batches Inside An Orchestration in BizTalk 2004

I think you could add the delay inside the .net component. Not sure what Thread.Sleep would do. But that might work.

11/20/2006 7:33 PM | Stephen W. Thomas

Stephen W. Thomas BizTalk Blog

My Links

News

Archives

BizTalk 2004 Samples

BizTalk 2006 Samples

BizTalk Videos

BizTalk White Papers

BizTalkBlogs.com

Great BizTalk Blogs

Other Links