Geeks With Blogs

News
Charles Young

This article has been superseded by a new article.

 

In the last few weeks, we've encountered a set of problems when using custom pipeline components in Receive pipelines.   After some testing, I have managed to produce a fairly definitive description of the behaviour we have encountered.   I have no very compelling explanation for what we have found, but the problems are entirely repeatable, and it is obvious from looking at BizTalk newsgroups that we are not the only people who have suffered from these problems.

 

In essence, if your pipeline contains the Xml Disassembler component, you are significantly restricted as to what you can do in any pipeline component you place in a subsequent stage (the Validate and ResolveParty stages), especially if you decide to assign an inbound map to your Receive port.   If you don't have a disassembler component in your pipeline, then everything works as you might expect from the documentation and examples provided by Microsoft.   So far, I haven't completed any testing using custom disassemblers.

 

Let's take a simple scenario.   You create a custom pipeline that will process inbound XML messages.   The custom pipeline contains the Xml Disassembler in order to detect the type of XML message.  This is one of the most basic uses of the XML Disassembler, which appends a MessageType promoted property to the message (as well as a schema reference).  When this property is set, BizTalk enables any inbound map set on the port.  You then create a pipeline component that will process the XML data within the message, making some fine adjustments (perhaps replacing an existing text node with a new value).   In real life, you might well want to do this is a situation where you are using an envelope to disassemble a single XML message into multiple messages.  You therefore want to place your pipeline component downstream of the Xml Disassember.   Armed with a knowledge of pipeline components based on Microsoft's current documentation and examples, you confidently place your pipeline component in the Validate stage and deploy your BizTalk application.

 

Your pipeline probably won't work.  Let’s say that your custom pipeline component attempts to process the XML data stream directly.   Perhaps it attempts to write directly to the data stream provided by the Xml Disassembler.  This certainly won't work.  The Xml Disassembler wraps its data stream in an instance of a class called XmlDasmStreamWrapper.   The underlying stream is an instance of a class called ForwardOnlyEventingReadStream.   The name of this class indicates fairly clearly why things won't work.   The stream is non-seekable (forward-only) and read-only.   You can't write to the stream and, having read the stream, you cannot reposition the stream pointer to the beginning of the stream again (e.g., by using myStream.Seek(0, SeekOrigin.Begin);).  If you read the stream, the map will fail, presumably because the stream is no longer positioned at the beginning of the data.

 

In passing, note that the general convention for pipeline components is that while the underlying stream is provided by calling GetOriginalDataStream() on a message part, you can obtain a second cloned stream by using the Data property.   Unfortunately, the Xml Disassembler does not comply with this convention.  The same stream is returned by both approaches.   I hunted around in the hope that the stream class might be storing the steam data in a private byte array that I could read through reflection.   However, unfortunately, the only thing I found was a buffer (the XmlDasmStreamWrapper class builds a buffer around the base stream), which of course is only filled if you read the stream.   In short, there doesn't appear to be any way getting at the data, let alone changing it, without adjusting the position to a non-zero value, and if you do this, your inbound map will break.

 

Now, let’s say that you change your custom pipeline component.   You read the entire stream into a string, process the string and then write the string out to a new, seekable, stream.   You reposition at the beginning of the stream and assign this stream back to the body part of the message.   This is one scenario that might possibly work.   As long as you don't have an inbound map, the amended message will make it through to the message box.   However, if you do have an inbound map on your Receive port, this will fail with a 'The root element is missing" error.   This error typically indicates that a stream containing XML has been read, and is not positioned at the beginning of the stream.   Hence, my best guess is that the inbound map is attempting to process the stream provided by the Xml Disassembler, and is not using the new stream.   The original stream has been read, and is positioned at the end of the stream.   Because it is non-seekable, there is no way you can reposition back to the beginning of the stream.  [Update - 29th Sept:  My 'best guess' turned out to be incorrect.   The map is using the stream provided on the body part of the message returned by the last pipeline component, as you might expect.   The issue is that the map will transform data sucessfully if this stream is non-seekable.   If it is seekable, BizTalk fails as described.   This appears to be what my 12-year old daughter might call a 'random' er.. feature (as in "oh, Dad, you're so random"].

 

Now, to see how truly bizarre things get, let’s imagine a slightly different scenario.   Let’s say that your pipeline component, for some reason, does not process the stream provided by the Xml Disassembler, but instead creates a new stream from scratch.   The important point here is that the original stream remains unread.   Based on the previous paragraph, you might expect that an inbound map would now work.   This is not the case, however.   Instead, regardless of whether you have an inbound map or not on your Receive port, your pipeline will fail.  In fact, BizTalk will pretty much lock up.  The BizTalk engine enters a never-ending loop in which it repeatedly re-executes the Validate and ResolveParty stages of your custom pipeline.   The only way out of this is to kill the BTSNTSvc.exe service and then quickly un-deploy your BizTalk assembly containing the custom pipeline.   If you don't un-deploy, the BizTalk service will be automatically re-started after a few seconds and will re-enter the closed loop.  [Update - 28th Sept:  I believe I can now explain this behaviour, and have added a feedback comment discussing why I believe this happens.   It appears to be a 'feature' of the XMLDisassembler component and the XmlDASMReader class used by XmlDasmStreamWrapper].

 

You can easily suppress this behaviour.   All you have to do is make sure that you have read the original stream provided by the Xml Disassembler.   Of course, if you do this, your inbound map won't work (you'll get the "The root element is missing" error).   However, if you don't have an inbound map, a message will be placed in the message box containing the contents of your new stream.

 

So there you have it.   If any of this is by design, then it rather suggests that Microsoft's intention is that, having used the Xml Disassembler, you should treat the body part data stream as immutable. [I'm less and less convinced any of this is 'by design']   This makes some sense, as the Xml Disassembler has specified the XML type within the message context, and you might inadvertently invalidate this information by any changes you make.   However, if this is intentional, it should be documented.   There is no hint anywhere that I can find in the existing documentation that this a design feature.   The closed-loop behaviour feels like a bug. [again, see the feedback comment below, which discusses this issue further]  I must admit that, generally speaking, you would not want to replace an unread stream provided by a disassembler component with a new stream.   However, neither should we have a situation where the BizTalk pipeline disappears forever into a black hole.

 

If you want to adjust the contents of a message after the disassemble stage, and also use an inbound map, you must either use functoids within your map to make the adjustments, or make the changes in a pipeline component and move your map into an orchestration.

 

You might be wondering about Microsoft's XML Validation and Party Resolution components.   I spent some time looking at these, and can confirm that neither of these components actually reads the body part data stream.  The validator, for example, simply wraps the unread stream in an XMLValidatingStream instance.   The validation is therefore done when the streeam is next read (probably by the the inbound map).  The Party Resolution component doesn't read the stream either.  It only changes the message context.

Posted on Thursday, September 23, 2004 10:56 AM | Back to top


Comments on this post: BizTalk Server 2004: Receive pipeline woes

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
Thanks for this nice information with good explanations of what's going on under the hood. :)

I would not recommend having maps in the inbound ports:
"especially if you decide to assign an inbound map to your Receive port."

The problem with inbound maps is that BizTalk has a bug or missing feature that makes inbound maps not exportable/importable through the BTSDeploy tool. I have worked around this having an extra receive port/location:
RecLoc1 -> SendPort1 (with first map) -> stage-file -> RecLoc2 -> SendPort2 .
Of course you can have two maps in the send port, but this
Left by Niklas E on Sep 23, 2004 11:40 PM

# Receive Pipeline woes
Requesting Gravatar...
Charles Young has a post explaining a problem with Receive Pipelines in BizTalk Server 2004 that contain the Xml Dissasembler...
Left by Commonality on Sep 23, 2004 11:08 PM

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
Hi Charles,

In our project we extensively used pipeline components in receive pipelines, but we do not have any BizTalk maps in our receive ports. For us it was not flexible enough to change these maps.
But nevertheless we have some experiences with pipeline components that i will try to describe.
When reading your blog entry my first idea was that you did not created a new IBaseMessage after changing the stream. We had some discussions with the guys in Redmond and they recommended us always to create a new IBaseMessage when reading and/or writing to the stream, indepentent if the component will be executed before or after disassembling stage.

We used this method:
protected IBaseMessage CloneMessageWithoutContext(IPipelineContext pc, IBaseMessage inMsg)
{
IBaseMessage outMsg = null;
IBaseMessageFactory msgFactory = null;
IBaseMessagePart outMsgPart = null;
IBaseMessagePart inMsgPart = null;
string partName = string.Empty;
bool isBodyPart = false;

msgFactory = pc.GetMessageFactory();
outMsg = msgFactory.CreateMessage();
for(int i=0; i<inMsg.PartCount; i++)
{
inMsgPart = inMsg.GetPartByIndex(i, out partName);
outMsgPart = msgFactory.CreateMessagePart();
outMsgPart.PartProperties = inMsgPart.PartProperties;
outMsgPart.Charset = inMsgPart.Charset;
outMsgPart.ContentType = inMsgPart.ContentType;
outMsgPart.Data = inMsgPart.Data;
outMsgPart.PartProperties = outMsgPart.PartProperties;
if(inMsgPart.PartID == inMsg.BodyPart.PartID)
{
isBodyPart = true;
}
else
{
isBodyPart = false;
}
outMsg.AddPart(partName, outMsgPart, isBodyPart);
}
return outMsg;
}
If you want to have all properties out of the context of the original message in the new message you can easily assign the context (no copy needed):
outMsg.Context = msg.Context;
I would assume that some properties (message type, port name, etc.) will be used by the BizTalk messaging engine to determine the map to be executed.
In the Execute-method you have to assign the changed stream to the cloned IBaseMessage and return it instead of the original IBaseMessage.

The XmlDisassembler works in a lazy mode, that means it only reads the next chunk if a downstream pipeline component or the BizTalk messaging engine itself (when writing to the message box) pulls the stream. The big advantage of this behaviour is that the memory footprint is always very low, indepentent if the message is large or small. But this behaviour has some more consequences that are important. A common use of the XmlDisassembler is promoting proterties out of the content of the xml stream. All properties promoted by the XmlDisassembler can be accessed at the earliest after the complete inbound stream was read by the XmlDisassembler.

I hope this will help you. And by the way i am enjoying to read your blog, you have always interesting entries in it. Thank you...
Left by André Dammeyer on Sep 25, 2004 2:25 PM

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
Thanks Andre. Yes, I have tried code pretty much as above, and still had problems. Originally, I ran into problems trying to create a diagnostic pipeline component, and at the time I found it difficult to work out a consistent pattern to the behaviour I saw. I therefore ended up wiritng a 'matrix' of 16 different pipeline conmponents, each one very minimalist, taking a slightly different approach re. message, part and stream creation. 8 of these components created new IBaseMessages using the message factory pretty much exactly as above. Four of these components create new message parts, again just about exactly as in your code, while the other four simply added the body part of the in message to the new IBaseMessage. Within each group of 4, I used various combinations of creating new streams, cloning existing streams and using the existing streams directly.

The other 8 components did much the same, but amending and returning the in message directly, rather than creating a new message.

I could not detect any variation in behaviour based on creating new messages or body parts vs. reusing existing messages and body parts. All the variation of behaviour seemd to be driven simply by what I did with streams, regardless of what messages and parts those streams were assigned to.

I do agree strongly with the approach recommended by the Redmond guys. As a general rule, I believe that a pipeline component that reads or changes message part streams in any way should always create and return a new IBaseMessage. This is a clean approach which helps avoid unexpected side effects and other problems. However, my tests suggested that this approach is not strictly necessary, and does not affect the behaviour we observed.

Re. the XmlDisassembler's lazy mode - yes, I picked this up as well. At one point I even tried using reflection to inspect the contents of the component's buffer while trying to find a way to get at the data without adjusting the stream position, so I have seen this lazy mode in action and up close.
Left by Charles Young on Sep 27, 2004 2:22 AM

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
I found an exellent article of Christof Claassens about pipeline copmonents.

He explains something about reading the streams wich could explain why some messages dissapear.

http://www.microsoft.com/belux/nl/msdn/community/columns/claessens/custompp.mspx

I could be completely wrong here but i guess one of the problems has something to do with this.....

Even if I am wrong it still is an exellent article
Left by Patrick Wellink on Sep 28, 2004 6:55 AM

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
It certainly is a really excellent article. Thanks for the link. The article does indeed provide ideas on a possible way of overcoming some of the problems we’ve encountered. You could ensure that any pipeline component used after the XmlDisassembler provides a specialised stream class that internally reads a clone of the stream provided by the XmlDisassembler, and uses the threading techniques discussed to ensure that the cloned data is read by a second thread which is blocked until the original stream data has been read by another (BizTalk) thread. I’m fairly certain it this technique would avoid some of the problems. However, I have a suspicion that it would still not solve them all. Imagine a pipeline component that reads the stream from an XmlDisassembler and produces a new, amended stream with an amended version of the contents (this is exactly what a colleague of mine was attempting to do recently; see http://realise-systems.net/blog/jsaull/archive/2004/09/23/497.aspx). Based on the behaviour we have encountered, I have a suspicion that while this technique might allow an inbound map to function, such a map would end up transforming the data provided by the XmlDisassembler, rather that the amended data.

In the test code I wrote, I kept the code as simple and minimal as possible, and this included doing things like (in some of my 16 components) reading the entire stream provided by the XmlDisassambler and writing the contents out to a MemoryStream (effectively buffering the entire stream content in memory) which I then returned from the test pipeline component (having, of course, reset the pointer to the beginning of the stream). This is not a good technique to use in 'real' pipeline components, but it is still reasonable to expect this to work (it does work fine, if you don't have an XmlDisassembler). The really peculiar behaviour, which I don't yet understand, is that if I left the stream provided by the XmlDisassembler unread, and created a new completely new stream in the test component, BizTalk would always go into a closed loop, repeatedly re-executing the stage containing my test component. Effectively, I was abandoning the stream provided by the XmlDisassembler.

It's worth noting the behaviour of the Validation component provided by Microsoft. Although this component leaves the stream handed to it unread, it does not abandon this stream, but wraps it in an XmlValidatingStream. So, when this XmlValidatingStream is eventually read, BizTalk is actually reading the underlying stream provided by the XmlDisassembler.

To prevent the looping behaviour, I discovered (by trial and error) that all I needed to do was ensure that the XmlDisassembler stream was read by my test components, even though (some of) these component then discarded the contents of the stream and created a whole new stream operating on different data. This solved the looping problem, but in-bound maps still failed in this scenario, as if the inbound map is attempting to use the XmlDisassembly stream (rather than some new stream created and returned by a subsequent pipeline component), but finds that the stream has already been read, and the stream position is therefore at the end of the stream, rather than he beginning.

As I suggested in the article, it is as if the use of the XmlDisassembler ends up imposing an undocumented contractual obligation on a Receive Pipeline to use the stream provided by the XmlDisassembler component within an in-bound map, and also strongly discourages (but does not ultimately disallow) the creation of new streams to replace the body part stream provided at the disassembly stage.

What I don't know yet is if this behaviour is a) peculiar to using the XmlDisassembler (probably not), b) a more general feature related to the use of the Disassasembly stage or c) related possibly to the creation of MessageType and/or SchemaReference context properties.
Left by Charles Young on Sep 28, 2004 11:46 AM

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
I've spent a little more time poking around in the code for the XmlDisassembler, and I think I may now know why I get the looping behaviour described in the article. The XmlDisassembler component uses a stream reader class called XmlDASMReader. This has a property called IsNewDoc used to indicated when the reader has found a new document for disassembly. When BizTalk calls the Diassemble() method of the XmlDisassembler component, this flag is set to true. BizTalk then calls GetNext() which returns the XmlDasmStreamWrapper containing the contents of this new document. However, it seems that the 'new document' flag is only re-set when this XmlDasmStreamWrapper stream is subsequently read. If the stream is never read, the flag remains true. The logic implemented in the GetNext method is such that the next time BizTalk calls GetNext, a second XmlDasmStreamWrapper is created over the same contents. In effect, if you never read XmlDasmStreamWrapper, the GetNext method will never return a null, and BizTalk will enter a never-ending loop, repeatedly submitting the same content again and again to the message box.

I can easily envisage a requirement where, after the disassembly stage, a developer would want to introduce a pipeline component that, based on some criteria, discards the body part content for certain messages and generates some new content. If you ever find yourself writing such a component, don't forget to read the stream before discarding it!!
Left by Charles Young on Sep 28, 2004 7:10 PM

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
I'm on a bit of a roll with this one - Having yesterday worked out why the looping behaviour occurs, I can now shed more light on why inbound maps fail.

It turns out that, very simply, if an inbound map operates on a seekable stream, the map fails and a "The root element is missing" error is logged. If the stream is non-seekable, everything works fine.

The XmlDisassembler returns a non-seekable body part stream. In several of my test components, I created new streams using the .NET MemoryStream class. MemoryStream is seekable, and so the inbound map failed.

It is easy to investigate this behaviour by creating a pipeline component that uses a bespoke Stream class (you can derive it directly from System.IO.Stream if you wish). Implement the stream class (my bespoke stream simply wrapped a MemoryStream) and override the CanSeek property to return false. When you run a message through the receive port, the in-bound map will work fine.

Now change the code to return true from the CanSeek property, redeploy and run again. This time the inbound map will fail.

The only visible difference between the two situations is that when the stream is readable, BizTalk makes an additional call to the Length property. I tried varying the stream output encoding between UTF-8 and Unicode, and playing with the Length property (either doubling or dividing by 2, as relevant). Nothing made any difference to the behaviour.

I shall be creating a new blog article to describe my discoveries.
Left by Charles Young on Sep 29, 2004 9:16 AM

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
Charles, as an ex-Brit now in the States, I suggest you may want to use the term "custom" instead of "bespoke." It will have a lot of Americans scratching their heads....not known here! They will sort of guess what it means but not be quite sure.
Fascinating discoveries on pipelines - thanks.
Left by John Bonavia on Oct 27, 2004 4:40 PM

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
Thanks John. Will do a search and replace on both articles. Two nations forever divided by a common language.... Ah well, it makes our dialogue more colourful.
Left by Charles Young on Oct 28, 2004 10:01 AM

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
Hi Charles, I am facing problem while configuring multiple maps in outbound maps of receive port.

Some of maps disappears while configuring 2nd map. Please help me out

Thanks in advance

Prabha

Left by Prabha on Mar 11, 2005 1:26 PM

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
Is this link now broken ?

http://www.microsoft.com/belux/nl/msdn/community/columns/claessens/custompp.mspx

Anyone have a cached copy, or a new link ?

thanks


Left by Barry Wimlett on Apr 15, 2008 4:38 PM

# re: BizTalk Server 2004: Receive pipeline woes
Requesting Gravatar...
Hi, I believe I am experiencing an issue when attempting to map the xml structure received within a validation component, to an internal canonical structure when the incoming xml document has dateTime values.

When a new stream is created to replace the bodypart stream, dateTime values (as a BST string +01:00) that were promoted from the original incoming stream they get automaticly converted to UTC. As part of the creation of the canonical structure the dateTime values are converted to UTC manualy via the c# ToLocalTime() function.

The problem is that when accessing a promoted DateTime in the orchestration, the time value ends up being 2 hours behind BST instead of 1 hour being UTC.

Stream source = inmsg.BodyPart.GetOriginalDataStream();
if (source.CanSeek)
source.Position = 0;

MemoryStream ms = new MemoryStream();
PipelineHelper.CloneStream(source, ms);
ms.Position = 0;

string messageType = inmsg.Context.Read("MessageType", "http://schemas.microsoft.com/BizTalk/2003/system-properties").ToString();

Myapp.BizTalk.MessageFactory.MessageMapper factory = new Myapp.BizTalk.MessageFactory.MessageMapper();
Myapp.BizTalk.MessageFactory.ITransform obj = factory.GetObject(messageType);

MemoryStream outMS = (MemoryStream)obj.Canonicalise(ms);
outMS.Position = 0;

inmsg.BodyPart.Data = outMS;

BizTalk also appears to ignore DateTimes formatted with a Z and such values get treated as if they were localTime. .NET DateTime serialisation does not have this flaw nor does DateTime.Parse().
Left by Terry Carvin on Apr 06, 2009 6:49 PM

Your comment:
 (will show your gravatar)


Copyright © Charles Young | Powered by: GeeksWithBlogs.net