Geeks With Blogs
Edmund Zhao's BizTalk abc We share, We learn

A follow developer got “Out of memory” error on BizTalk server and he forwarded me the solution he build and the data he used to test.

 

The solution is pretty straightforward. He needs process about 50 flat files at same time. In average, the file size is about 2MB. Each file contains 8 sections and each section has multiple items with same format. The purpose of this solution is to selectively insert those data into a SQL database.

 

The solution has 2 steps. First step, get the file through a pipeline and convert it to an XML file, then get the XML file into an orchestration and use 8 maps in parallel action to map them to 8 XMLs representing 8 sections, the schema of those XMLs match the SQL request schema which will be used in step2, and then those XMLs will be dumped to local directories. Step 2 will go pickup the messages from local directories and send them directly to SQL adapter to update the SQL database.

 

I found there were lots of problems in this solution. But here I will only cite some major ones.

 

  1. The solution is applying several mappings directly on large messages (average size of 2MB). The way BizTalk 2004 handle mapping logic in large messages is either to split them into small pieces or put the mapping logic to somewhere other than the mapping tool. It is painful BizTalk 2004 mapping tool lack the ability to process large message. When you map large messages, the memory consumption will easily be 10 times or even more of the sizes of themselves. During my testing of this solution, a 2MB file consumed about 50MB memories in step 1 of the solution. This is the main reason of the “Out of memory” failure.
  2. Current design creates couple of XMLs in step1 and uses them as SQL requests to insert or update database through SQL adapter in step 2.  This design has following down sides, when you use SQL adapter, you will have to generate schema for all input parameters even you only use a small percentage of them. This will force you to create large request XML messages which contain the mapping results of original data. Normally it could be fine, but if your have large messages to process, these request messages will also become bigger.
  3. The parameters in stored procedures are too long. The consequent is when you get a large XML to process you will find most of them are not the data you actually need. I checked few XML files after I finished step 1, and found the real data in most of them are only 20% of the file sizes.

 

To fix the issue above, I gave the following suggestion to my fellow developer.

 

  1. The scenario in this solution is almost pure message routing. There are no many data flow logic involved, so we can get rid of mapping and even the SQL adapter. We can write our own .NET assembly including a mapping helper and a db helper to handle them programmatically, and then call the assembly in orchestrations. By doing that, for sure we will get much better performance.
  2. Write a custom pipeline component to cut the original flat file into 8 pieces, each one contains a section. Then split those flat file sections into single unit XML messages through a flat file disassembler in pipeline. Then load the XML into orchestration and do the mapping, finally call the SQL adapter to update database. Correlation may need to be required in this solution.
  3. Last suggestion is simple. Just rename the parameters in stored procedures. Give short name, this could cut XML message size at least 50% off.   

 P.S.

 BizTalk 2006 is supposed to solve the large message mapping issue. The following statement is from Microsoft web site.

 

Large message transformation. In previous versions of BizTalk Server, mapping of documents always occurred in-memory. While in-memory mapping provides the best performance, it can quickly consume resources when large documents are mapped. In BizTalk Server 2006, large messages will be mapped by the new large message transformation engine, which buffers message data to the file system, keeping the memory consumption flat.

 

...Edmund Zhao

Posted on Wednesday, December 21, 2005 2:00 PM | Back to top


Comments on this post: Out Of Memory issue when processing large messages in BizTalk

No comments posted yet.
Your comment:
 (will show your gravatar)


Copyright © Edmund Zhao | Powered by: GeeksWithBlogs.net