The Ten Commandments
of pipeline stream processing
1. Thou shalt avoid reading thy streams within a pipeline.
The ideal general pipeline architecture is to allow BizTalk to 'pull' each stream once after pipeline processing is complete. If you need to process message content in a non-streaming fashion, consider implementing your logic in an orchestration, or possibly within functoids and script.
2. Thou shalt wrap thy streams in wrapper classes.
Avoid returning 'raw' MemoryStreams from your pipelines. Create your own stream wrapper classes. This will give you much more control over the precise behaviour of your streams. Stream wrappers allow you to implement your own 'Read' methods to process streams in a linear fashion when the stream is pulled.
3. Thou shalt not discard an existing stream without first reading it.
Because streams are processed during reads, pipeline code is often dependent on side-effects that occur when the 'Read' method is called. It is therefore generally sensible to always read a stream, even if you then discard it, perhaps replacing it with a new stream (I realise this conflicts a bit with commandment 1). The message returned by the XmlDisassembler, for example, assumes that the body part stream will always be read. If you don't read it, BizTalk will re-execute the GetNext method of the disassembler continuously.
4. Thou shalt remember the Sabbath and keep it holy.
You know it makes sense. Those intractable stream-processing problems you face at 5:30 pm on Friday evening will have resolved themselves in your confused brain by Sunday morning as long as you don't go within 300 meters (yards if you are American or, like me, a Brit over the age of 35) of a computer all day Saturday. Some fundamentalist types (for example, my wife) believe you shouldn't use Sundays to develop code either, but this is considered by most to be an extreme position.
5. Thou shalt not return seekable streams to the pipeline.
Those who use seekable streams will find their inbound maps don't work (a problem that will be fixed in SP1). In any case, seeking is a great evil within the streaming world of a pipeline. Just don't do it.
6. Thou shalt not return writeable streams to the pipeline.
Don't give succour and comfort to those foolish developers who may try to write to your streams. Be like Microsoft. Make it as difficult as possible to do anything but read your streams in a forward-only fashion.
7. Thou shalt always flush thy buffers after writing to a stream.
It's a bit like having to remember to release allocated memory in C++ or call Release on IUnknown. If you forget, you may spend ages trying to work out why everything breaks. Of course, your steam may not use buffers, but it’s better to play safe. And don't forget to wash your hands after every flush.
8. If thou introducest new data content, thou shalt implement thine own message part classes to return cloned data streams via the Data property.
For, verily, this valuable design pattern does not happen by magic! One thing though. If at all possible, try to avoid breaking the first commandment when implementing a cloning mechanism. Christof Claesson's article should be your guide here.
9. Thou shalt think hard and long before using an unknown implementation of the Data property of a message part.
Microsoft's StreamViewOfIStream class (used in messages submitted from native adapters) provides an example of code that 'breaks' the first commandment in order to provide a clone. This can have unexpected side-effects that are difficult to diagnose. This should not be regarded as a mistake, on Microsoft’s part, however…which leads us on to the final commandment…
10. Thou shalt not treat any of these commandments as sacrosanct, except, perhaps, the fourth.
These commandments are guidelines for fostering good pipeline component design. They are there to keep you on the straight and narrow until you reach BizTalk pipeline streaming maturity. Ultimately, you do not live under law, and should feel free to 'break' the commandments as necessary. However, you should not break any of them unadvisedly, lightly or wantonly, but advisedly, soberly and after much thought.
Lastly, note that these commandments do not address the issue of multithreading. The best pipeline component implementations, however, are likely to use multithreading techniques in order to maximise performance, throughput and scalability. But that’s another story…