Edmund Zhao's BizTalk abc

We share, We learn

  Home  |   Contact  |   Syndication    |   Login
  23 Posts | 1 Stories | 14 Comments | 8 Trackbacks

News



Article Categories

Archives

Sunday, September 04, 2011 #

If your orchestration is heavily rely on WCF SQL adapter based on SQL stored procedures, you may often encounter an issue related to strongly-typed schema and receive an error which is similar to the following.

The adapter failed to transmit message going to send port "WcfSendPort" with URL "mssql://sqlserver//database?". It will be retransmitted after the retry interval specified for this Send Port. Details:"System.Data.SqlClient.SqlException: Invalid object name '#Temp'.

This is because of the limitation of BizTalk Adapter Pack 2.0, it doesn't support temp table in stored procedure if you need to generate typed-procedure schema. You can use table variable to replace temp table, but it won't work if you use dynamic SQL, or the table is not small. But don't worry, there is a easy workaround, just make sure your stored procedure contain the following contents.

    --Ensure FMTONLY OFF just before creating temporary table
    DECLARE @CheckFmt bit;

    SELECT @CheckFmt = 0 WHERE 1=1

    IF @CheckFmt IS NULL
       SET FMTONLY OFF;

    --Create temporary table
    IF object_id('tempdb.dbo.#Temp') IS NOT NULL
       DROP TABLE #Temp;
   
    Create table #Temp(
     ...............
    )   
   
    --Ensure FMTONLY is turned back on if it was on before
    IF @CheckFmt IS NULL
            SET FMTONLY ON;

    --Feel free to do anything on the temp table here


    --Make sure you drop it at the end of your stored procedure
    IF object_id('tempdb.dbo.#Temp') IS NOT NULL
       DROP TABLE #Temp;

If you want to know more details about this limitation as well as some other interesting stuff about WCF SQL adapter, a very good article here is recommanded to you for reading.

...Edmund

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Friday, August 12, 2011 #

Performing Composite Operations on Oracle Database is very similar to SQL Server Database. I just want to point out one mistake developers might easily make during the send port configuration. For any composite operation, the generated binding script doesn't include the composition operation tag in SOAP action header, therefore a new line needs to be manually added for each composite operation during the send port configuration.

For WCF SQL Adapter, it looks like
 <Operation Name="CompositeOperationName" Action="CompositeOperation" />

But if you do the same thing when you configure WCF Oracle adapter, you will receive "Argument CompositeOperation is invalid" error in event log at runtime.

The right format for Oracle adapter should look like

 <Operation Name="CompositeOperationName" Action="http://Microsoft.LobServices.OracleDB/2007/03/CompositeOperation" />
     
The difference is for Oracle you need to add namespace before "CompositeOperation". It makes sense bacause this is a Microsoft product and SQL Server Adapter is the default adapter, so only SQL Server Adapter doesn't require namespace.

For a complete reference on how to performing composite operations on Oracle database could be found here on MSDN.

...Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Monday, July 25, 2011 #

This is a unknown issue with Certificates in BizTalk Server. You see the error every time when you open or save send port configuration, it won't affect the functionality, but very annoying. It usually happens when you remotedly log on to a BizTalk server at which the "Other People Certificate Store" is not initialized.

Microsoft has an article talking about this issue, and it suggests to initialized the "Other People Certificate Store" during the Group configuration. However, if you have your BizTalk group configured already and don't want to reconfigure it, here is what you can do,

1. Open "Internet Options" from control panel, select "Content" tab, and click "Certificates".
2. Click "Other People" tab, click "Import...", this will launch the import wizard. Click "Next" then Click "Cancel", because you don't really have a certificate to import.
3. You should go back to "Other People" tab now. Click "Advanced" button, check all check boxes, click ok to save it.

Good luck!
...Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Monday, June 27, 2011 #

Binding file for WCF Adapter doesn't save the password no matter it is generated by "Add Generated Items..." wizard in Visual Studio or "Export Bindings..." in administration console. It is by design dut to the consideration of security, but it is very annoying especially when you import bindings which contain multiple WCF send ports. The way to aviod retyping password everytime after an import is to edit the binding file before import. Here is what needs to be done.

1. Find the following string:

    &lt;Password vt="1" /&gt;

"&lt;" means "<", "&gt;" means ">", "vt" means "Variable Type", variable type 1 is "NULL", so the above string can be translated to "<Password/>"

2. Replace it with:

    &lt;Password vt="8"&gt;MyPassword&lt;/Password&gt;   

variable type 8 is "string", the above string can be transalted to "<Password>MyPassword</Password>"
 
Binding file uses a lot of character entity references for XML character encoding purpose. For a list of the special charactor entiy references, you can check from here.

...Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Sunday, June 12, 2011 #

One of the most common errors we could see when we use WCF adapters is "The action was not understood". This is most likely because the mismatch between the logical port operation identifier in a orchestration and  the operation name of SOAP action header in WCF send port confguration. To fix the problem is very easy, just change one of them to have them match.

There is a very good article explaining why this happens. Click here to view the article.

...Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Thursday, May 05, 2011 #

If you organize your BizTalk projects based on the type of artifacts, e.g., a solution contains schema project, pipeline project, map project, orchestration project, you might often encounter build failure problems due to the missing assembly reference. Sometime, even if you have the references properly added and you are able to verify them in object browser, you might still have problem not seeing the newly added or modified artifacts in any configuration window.

The reasons for this kind of issues could be various, but most time they are related to the modification of the assemblies being referenced. Theoretically if your assembly references are based on projects in same solution, Visual Studio should update the reference automatically when you rebuild the solution. But lots of time, it just not happened, especially after you regenerate generated items. A good example is the project using WCF adapters. My experience was, almost every time after I regenerated WCF adapter schemas, references among projects were broken more or less.

The following is what I recommend to completely cleanup and reestablish all the references among projects in case you cannot fix the reference missing issue quickly. It contains some steps unnecessary on certain situation, but saves much more time than keep trying all different ways to add the references back.

1. If BizTalk server is installed on the same server where the development environment is, stop the corresponding applications and terminate all related suspended or active items using administration console, then delete those applications.

2. Delete project related assemblies from GAC.

3. Delete all bin folders from application directories. Sometime, if certain dll is not removable, you may have to reboot your machine first then try to delete again.

4. Build the project one by one based on their build sequence. For example, you can build the schema project first, then remove and re-add the schema project reference in map project, then build the map project.

5. By doing the above, most projects should be built successfully. However, orchestration project build may still fail. If that's the case. Open all odx files in text editor, remove everything after '#endif // __ DESIGNER_DATA' (Those are the computer generated code, won't be updated automatically) from each file, save them, open them again in orchestration editor, make some slight changes, e.g., move the logical port a little bit, then save them again. This is to make sure code after '#endif // __ DESIGNER_DATA' will be regenerated based on the updated reference information, otherwise orchestrations won't be visible after deployement.

 ....Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Thursday, April 07, 2011 #

I've been through a scenario where I had a very complicated multiple schemas map, the left(source) side of the map had two schemas and the right(destination) side had one, for some reason I decided to add one more schema to the left side.

BizTalk map editor in 2006/2006 R2/2009/2010 doesn't allow creating multiple schemas map directly, neither does it allow adding or removing schemas from either side of the map, therefore in order to add a schema, I would have to open an orchestration, create a transform shape then create a new multiple schemas map in transform configuration window. However, by doing this, I would end up with a new blank map and it would be a big headache for me to put all the mapping info back and retest it.

There is a reason I really like BizTalk, because most BizTalk artifacts are literally just XML. Therefore, as long as I have a XML editor, or any other text editor, I can always open the file and modify it by myself.

Here is what I did, it's not difficult at all and took less than five minutes for me to complete and test.

1. Open the existing map file "multisourcemap.btm" using any text editor tool.

2. Find the node <SrcTree>, it is usually close to the begining of the file, should be easy to locate.

3. Modify the child node <xs:schema>. In my case, I had two source schemas in existing map already, so it looks similar to

<xs:schema xmlns:tns="http://schemas.microsoft.com/BizTalk/2003/aggschema" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" xmlns:ns1="http://SourceSchema1NameSpace" xmlns:ns2="http://SourceSchema2NameSpace" targetNamespace="http://schemas.microsoft.com/BizTalk/2003/aggschema" xmlns:xs="http://www.w3.org/2001/XMLSchema">

     Now I just need to add "xmlns:ns3" attribute and make it look similar to

<xs:schema xmlns:tns="http://schemas.microsoft.com/BizTalk/2003/aggschema" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" xmlns:ns1="http://SourceSchema1NameSpace" xmlns:ns2="http://SourceSchema2NameSpace" xmlns:ns3="http://SourceSchema3NameSpace" targetNamespace="http://schemas.microsoft.com/BizTalk/2003/aggschema" xmlns:xs="http://www.w3.org/2001/XMLSchema">

4. The <xs:schema> node has two <xs:import> child nodes already, it looks similar to

<xs:import schemaLocation="Schema1Location" namespace="http://Schema1NameSpace" />
<xs:import schemaLocation="Schema2Location" namespace="http://Schema2NameSpace" />

     So just add another one

<xs:import schemaLocation="Schema3Location" namespace="http://Schema3NameSpace" />

5. After <xs:import> node, there is a <xs:element> node, looks similar to

      <xs:element name="Root">
        <xs:complexType>
          <xs:sequence>
            <xs:element name="InputMessagePart_0">
              <xs:complexType>
                <xs:sequence>
                  <xs:element ref="ns1:Schema1RootNodeName" />
                </xs:sequence>
              </xs:complexType>
            </xs:element>
            <xs:element name="InputMessagePart_1">
              <xs:complexType>
                <xs:sequence>
                  <xs:element ref="ns2:Schema2RootNodeName" />
                </xs:sequence>
              </xs:complexType>
            </xs:element>
          </xs:sequence>
        </xs:complexType>
      </xs:element>
    </xs:schema>

    Add another section <xs:element name="InputMessagePart_2">, make it similar to

     <xs:element name="Root">
        <xs:complexType>
          <xs:sequence>
            <xs:element name="InputMessagePart_0">
              <xs:complexType>
                <xs:sequence>
                  <xs:element ref="ns1:Schema1RootNodeName" />
                </xs:sequence>
              </xs:complexType>
            </xs:element>
            <xs:element name="InputMessagePart_1">
              <xs:complexType>
                <xs:sequence>
                  <xs:element ref="ns2:Schema2RootNodeName" />
                </xs:sequence>
              </xs:complexType>
            </xs:element>
            <xs:element name="InputMessagePart_2">
              <xs:complexType>
                <xs:sequence>
                  <xs:element ref="ns3:Schema3RootNodeName" />
                </xs:sequence>
              </xs:complexType>
            </xs:element>
          </xs:sequence>
        </xs:complexType>
      </xs:element>
    </xs:schema>

6. Save the btm file.

That's it, the modfied map file can be opened in BizTalk map editor, new schema is added, and all old mapping info are still there!

....Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Monday, April 02, 2007 #

You might receive the "pipeline component load() method failed on IPersisPropertyBag implementation" error message when you try to drag and drop a custom pipeline component into a pipeline stage. The root cause of this issue can be various, but the most common reason for this kind of failure is because ArgumentException is not caught when you develop the Load method of IPersistPropertyBag interface in your custom pipeline component class. A common way to catch exceptions in Load method is similar to the following:

            catch(ArgumentException)
            {
                return val;
            }
            catch(Exception ex)
            {
                throw new ApplicationException(ex.Message);
            }

If the first catch is missed, you will receive error when Load method runs for first time.

...Edmund Zhao  

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Friday, March 09, 2007 #

In multiple BizTalk server environment, it is pretty common to re-register another BizTalk server on BAS site, and the process could be tricky sometime if you don't want to undeploy anything from BizTalk solution. The following are the steps need to be performed,

1. Open BizTalk Explorer -> Role -> Sender - Delete all
2. Go to TPM Management Tools -> http://localhost/BASSite/Lists/Agreements/AllItems.aspx and inactivate all agreements.
3. Go to TPM Management Tools -> Partner Profiles -> Undeploy all partners
4. Go to TPM Management Tools -> Unregister BizTalk Server (sts.outbox may be bind to orchestration already. In case of that, use BizTalk explorer to unbind it from corresponding orchestrations first. Check if sts.outbox has been deleted, if not, manually delete it)
5. Register BizTalk server
6. Bind sts.outbox to orchestrations
7. Deploy all partners
8. Activate all agreements
9. Check Role -> Sender to see if the party has been enlisted or not

...Edmund Zhao

 

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Tuesday, February 13, 2007 #

I received a very unusual error message few days ago when I configured a secondary BizTalk Server to join an existing server group (BizTalk 2004). The error message said "The BAM Primary Import Database registered in the BizTalk Management database is not compatible with the currently installed version of BizTalk Server." It happened only when I added the server to this particular server group, there was no error when I chose to join other server groups. Apparently something was wrong in the configuration of this group.

Before taking action to re-configure the master BizTalk server in that group (of course, all BizTalk applications also need to be redeployed), I checked the database carefully. I found the table "BizTalkDBVersion" in BAM Primary Import Database was empty. So I decided to give a try by manually insert a record into this table. I grabbed the version parameter from corresponding tables on other servers, and inserted a record. After all, the BizTalk configuration allowed me to use this Management batabase!

This helped me to understand the role "BizTalkDBVersion" table plays in each BizTalk databases.

...Edmund Zhao  

 

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Tuesday, October 31, 2006 #

Does master secret server always have to be clustered? My answer is practically it is not. The environment I'm working on has 8 BizTalk servers in each server group. Master secret server is on one of these BizTalk servers, not being clustered. The following is how to have the Biztalk server groups to be recovered in case any failure of the master secret server.

1. Make a backup of the secret on master secret server, before it crashes! And store a copy of the secret on each BizTalk server.

2. When the master secret server fails, follow the steps below.

(a) Update "GlobalInfo.xml" under "C:\Program Files\Microsoft BizTalk Server 2004\SDK\Samples\SSO\Manage" and save to "C:\Program Files\Common Files\Enterprise Single Sign-On". And make sure you have the new master secret server name in <secretServer>New Master Secret Server</secretServer> tag

(b) Go to "C:\Program Files\Common Files\Enterprise Single Sign-On". Run "SSOmanage -updatedb GlobalInfo.xml". You might have to go to different directory if you didn't use default directory for BizTalk installation

(c) Run "SSOmanage -serverall <new master secret server>"

(d) In BizTalk admin console change the Enterprise Single Sign-on Server name to <new master secret server>. It is under server group properties -> General tab

(e) Run "SSOmanage -displaydb" to verify the information in the SSO db

(f) Restart ENTSSO service on the new master secret server

(g) Run "SSOconfig -restoresecret <master secret key backup>"

(h) Verify the change by doing health check in BizTalk admin console.

...Edmund Zhao 

 

   

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Monday, September 11, 2006 #

It is usually a very important task to clean up a BizTalk mseeage box during stress testing with large amount of messages. Below are the steps to clean up the message box in order to prepare for the new test.

1. Stop all BizTalk services

2. Type “iisreset” at command line to recycle IIS service

3. Execute the stored procedure “bts_CleanupMsgbox” on your message box database

4. Execute the stroed procedure “bts_PurgeSubscriptions” on your message box database

(if the above stored procedures can't be found, go to your BizTalk installation directory, under “Schema” folder, you can find all original scripts of these stored procedures)

5. Clean up the message box log by running the backup statement similar to below

backup log msgBoxDb to disk = “yourbackupdirectory\yourbackupfile.bak' with init, stats = 5

6. Restart BizTalk services

...Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Monday, March 06, 2006 #

There are two types of dynamic ports in BizTalk 2004. They are Dynamic One-Way Port and Dynamic Solicit-Response Port. The following sample is based on Dynamic One-Way Port, using the other one will be very similar.

1. Create a new BizTalk project.
2. Create a schema to the new project.
   
3. Promote “Destination” to be a distinguished field.
4. Create an orchestration to the project. “DynamicSendPort” is a configure port with “Dynamic” port binding.
   
5. Add the following statement into expression shape. “MsgSample” is the message created in orchestration view
          DynamicSendPort(Microsoft.XLANGs.BaseTypes.Address) = MsgSample.Destination;
6. Associate the strong name key to the project and deploy it.
7. Create receive port in BizTalk Explorer pointing to a local file directory.
8. Bind the receive port to orchestration and start the orchestration.
9. Drop the XML instance file to the receive location and you will see the message sent to the location defined in the instance file. The XML instance file should look like the following.

 
Destinations could be on different protocols as long as the URI is correct. BizTalk will generate “The outbound transport could not be resolved” error if URI couldn't be found. The following are some valid formats.
    file://c:\file\Dynamic\Output\%MessageID%.xml
    msmq://.\private$\TestQueue
    http://orders.mycompany

…Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Wednesday, February 08, 2006 #

I passed the exam 74-135 yesterday. It was really tough, might be one of the most difficult exams provided by Microsoft. The reason I took this exam was because my company wanted to join BizTalk 2006 Jump Start program, and one of the requirements to participate the program was at least having a consultant passed 74-135. I spent almost a month to prepare for the exam, and finally passed it with all sectional results marked as “Strong”.

If you are also going to take the exam, here are some background information and tips for your preparation.

Exam: 74-135
Name: Developing E-Business Solutions Using Microsoft BizTalk Server 2004 
Questions: 40
Time: 130 minutes
Passing score: 700 out of 1000

1. If you are very family with BizTalk 2004 and have few year experiences, you don't have to worry too much, I'm sure you will pass it. However, I would still suggest you spend few weeks to prepare for it, mainly to fill some gaps between your hands-on experiences and the exam required knowledge, because they will ask you all kinds of questions, even the almost retired HWS!
2. If you are new to BizTalk 2004 or only did some MOC or few simple BizTalk projects so far, I would suggest you to fully prepare for it before you go take the exam. Please refer to my blog on Nov 29, 2005 to see all the resources you could use for preparation. There is no way to pass this exam without a very good understanding of BizTalk 2004!
3. Practice, practice and practice. Like others mentioned in their blogs after they took the exam, lots of practicing always help.

...Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Wednesday, February 01, 2006 #

BizTalk Server 2004 is not cluster aware, therefore it is not recommended to install BizTalk 2004 on a MSCS based server cluster. However, to achieve high availability of you BizTalk solution, you can still take the following actions. (I assume you are installing BizTalk 2004, SQL Server 2000 on Windows Server 2003)

1. Setup an Active\Passive server cluster based on Microsoft Cluster Services
Install SQL Server 2000 with service pack 3a on the cluster. This step is very easy because SQL Server 2000 fully support MSCS. The following is a great article from Brad M. McGehee telling you all the details. The article was talking about intalling SQL server 2000 on Windows Server 2000, not Windows Server 2003, but it makes no big difference.

2. Install SQL Server 2000 Analysis Services with service pack 3a on the cluster. This step is a little bit complicated because Analysis Services is not cluster-awared. Lots of manual processes will be involved during installation. The following is the detail instruction of installation. The instruction doesn’t mention when and how to install service pack 3a. You need to be ware of that and try to apply the service pack before manually modify things like registry.

3. Install Microsoft Single Sign-On Master Secret Server. This step won’t be too complicated if you have already experienced the installation of Analysis Service cluster. You can follow the instruction below in a great article provided by Mike H.

4. Test them by moving groups in Cluster Administrator

...Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Friday, January 06, 2006 #

So far, the most complete document about BizTalk 2004 naming conventions is still in the MSDN article “Developing Integration Solutions with BizTalk Server 2004”. However, if you want to adopt these naming conventions, you need to be aware of the following issues in the document,

 

  1. Naming conventions for some BizTalk artifacts are not defined. For example Port Operations, Send/Receive Ports…
  2. Some naming conventions are unpractical. For example, the naming convention for Map file is “_To_.btm”. But in a real world, if you have multiple source schemas or destination schemas, the name will be too long.
  3. Some naming conventions will cause duplication. For example, the naming convention for Send Shape is “Snd_”, how if you send same message a couple of times?

 

Having a strict artifact naming convention on all stage of a BizTalk project could be a disaster sometime. It could significantly slow down the development especially when you need to redo something like an orchestration. So my suggestion is to do minimum custom naming during development phase, and rename them later when your solution becomes mature enough. By saying minimum custom naming I mean you still need to give a proper name to an artifact at the very beginning if renaming later could cause lots of corresponding changes in your solution.

 

...Edmund Zhao    

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

One important pattern in BizTalk is to partitioning the solution especially when it is big. But some time it’s really difficult to move a file to another project or even just a different folder in same project. This happens most likely when there is an automatically generated file involved.

 

In this example, an orchestration needs to consume a web service. So we added a web reference in the project contained the orchestration. Then we built a map file which maps a schema to the request schema in Reference.xsd which was automatically generated when you added the web reference. In the orchestration, we used the map to build a request message and called the web service.

 

Now we need to move the map file from the orchestration project to a separated map project. But here is the problem. Map project needs to refer to orchestration project because Reference.xsd is still in orchestration project, but orchestration project also needs to refer to the map project because orchestration uses a map in map project. How do we solve this cross reference issue?

 

Here is a solution to this issue.

  1. Still add web reference in your orchestration project.
  2. Copy the automatically generated schema Reference.xsd to another schema project. (This break the referral of your orchestration project, notice every time you change the web service signature, you will need to make a new copy)
  3. Build the map in your map project referring to the schema copied in schema project.
  4. Modify the orchestration. Add one more message based on the Reference.xsd in schema project. After mapping, do a “Message Assignment” to convert your message from external schema to internal schema.

 

This approach has a limitation that is you will have to use “Microsoft.BizTalk.DefaultPipelines.PassThruTransmit” to skip namespace check in your send port. If you use “Microsoft.BizTalk.DefaultPipelines.XMLTransmit” pipeline, an error “There was a failure executing the send pipeline…” will occur.

 

...Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Wednesday, December 21, 2005 #

A follow developer got “Out of memory” error on BizTalk server and he forwarded me the solution he build and the data he used to test.

 

The solution is pretty straightforward. He needs process about 50 flat files at same time. In average, the file size is about 2MB. Each file contains 8 sections and each section has multiple items with same format. The purpose of this solution is to selectively insert those data into a SQL database.

 

The solution has 2 steps. First step, get the file through a pipeline and convert it to an XML file, then get the XML file into an orchestration and use 8 maps in parallel action to map them to 8 XMLs representing 8 sections, the schema of those XMLs match the SQL request schema which will be used in step2, and then those XMLs will be dumped to local directories. Step 2 will go pickup the messages from local directories and send them directly to SQL adapter to update the SQL database.

 

I found there were lots of problems in this solution. But here I will only cite some major ones.

 

  1. The solution is applying several mappings directly on large messages (average size of 2MB). The way BizTalk 2004 handle mapping logic in large messages is either to split them into small pieces or put the mapping logic to somewhere other than the mapping tool. It is painful BizTalk 2004 mapping tool lack the ability to process large message. When you map large messages, the memory consumption will easily be 10 times or even more of the sizes of themselves. During my testing of this solution, a 2MB file consumed about 50MB memories in step 1 of the solution. This is the main reason of the “Out of memory” failure.
  2. Current design creates couple of XMLs in step1 and uses them as SQL requests to insert or update database through SQL adapter in step 2.  This design has following down sides, when you use SQL adapter, you will have to generate schema for all input parameters even you only use a small percentage of them. This will force you to create large request XML messages which contain the mapping results of original data. Normally it could be fine, but if your have large messages to process, these request messages will also become bigger.
  3. The parameters in stored procedures are too long. The consequent is when you get a large XML to process you will find most of them are not the data you actually need. I checked few XML files after I finished step 1, and found the real data in most of them are only 20% of the file sizes.

 

To fix the issue above, I gave the following suggestion to my fellow developer.

 

  1. The scenario in this solution is almost pure message routing. There are no many data flow logic involved, so we can get rid of mapping and even the SQL adapter. We can write our own .NET assembly including a mapping helper and a db helper to handle them programmatically, and then call the assembly in orchestrations. By doing that, for sure we will get much better performance.
  2. Write a custom pipeline component to cut the original flat file into 8 pieces, each one contains a section. Then split those flat file sections into single unit XML messages through a flat file disassembler in pipeline. Then load the XML into orchestration and do the mapping, finally call the SQL adapter to update database. Correlation may need to be required in this solution.
  3. Last suggestion is simple. Just rename the parameters in stored procedures. Give short name, this could cut XML message size at least 50% off.   

 P.S.

 BizTalk 2006 is supposed to solve the large message mapping issue. The following statement is from Microsoft web site.

 

Large message transformation. In previous versions of BizTalk Server, mapping of documents always occurred in-memory. While in-memory mapping provides the best performance, it can quickly consume resources when large documents are mapped. In BizTalk Server 2006, large messages will be mapped by the new large message transformation engine, which buffers message data to the file system, keeping the memory consumption flat.

 

...Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Wednesday, December 14, 2005 #

Lots of BizTalk developers were facing similar problems when they built their own .NET test applications to test the web services generated by BizTalk Web Services Publishing Wizard. The debugging process stopped at “this.Invoke()”, and usually there will be an error message of “Internal SOAP Processing Failure”. If you check the event log, you will also see an error similar to “The Messaging Engine failed to register the adapter for SOAP for the receive location ...asmx. Please verify that the receive location is valid, and that the isolated adapter runs under an account that has access to the BizTalk databases.”

 

The first thing you can do is to check the status of your orchestration in BizTalk Explorer. Make sure the orchestration is properly bound to its corresponding web ports, and it is enlisted and started. If the status of your orchestration is ok, most likely it’s a security issue on the machine you are running the web services and BizTalk orchestration.

 

Here are some tips to help you solve the issues.

 

Q: Should I turn off anonymous access to the web service project?

A: No, otherwise you will receive “The request failed with HTTP status 401: Access Denied.” error.

 

Q: Should I change the default namespace “http://tempurl.org/” of my web service to match the real URL of the web service?

A: No, it’s not necessary. But it will be a good practice to rename it to be something meaningful.

 

Q: I’m running my test on Windows 2003 and I received “Internal SOAP Processing Failure” error. What should I do?

A: Find out the identity under which the application pool is running. Add this user to both “IIS_WPG” and “BizTalk Isolated Host Users” groups. Check the permission of %temp% folder. Make sure either the user running the application pool or the groups this user belongs to has read/write permission on %temp% folder.

 

Q: I’m running my test on Windows XP and I received “Internal SOAP Processing Failure” error. What should I do?

A: Add user “ASPNET” into “BizTalk Isolated Host Users” group. Check the permission of %temp% folder. Make sure either ASPNET or “BizTalk Isolated Host Users” group has read/write permission on %temp% folder. If your %temp% folder is on FAT32, don’t worry about the permission.

 

A good reference for you to understand more details is an MSDN article BizTalk Server 2004 and Web Services.

 

...Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati

Tuesday, November 29, 2005 #

I have been working on BizTalk for few years since the 2000 version was released. Although Microsoft did some great job to improve their online technical information about BizTalk during the past years, I believe lots of BizTalk developers are still using groups and blogs as a preferred resource of information. BizTalk is such a detailed technology that it will be very hard sometime for a developer to find a proper pattern to solve a particular problem. That's why I'm going to build this blog and start to do something for our community. I'd like to thank every fellow BizTalk blogger, your wonderful contribution made my job much easier.

 

...Edmund Zhao

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati