Bram Veldhoen

Debatching, aggregation and resequencing using the BizTalk ESB Toolkit 2.0 and ESB.Extensions

Bram Veldhoen — Fri, 11 Sep 2009 12:24:44 GMT

The ESB.Extensions framework is out!! You can download it here.

The ESB.Extensions builds on top of the ESB Toolkit 2.0, and thus requires BizTalk Server 2009. The ESB.Extensions contains a number of services, that are itinerary-aware: ReceivePipelineService, SendPipelineService, ResequencerService and ResequencerGoService. Let's look at them more closely.

The ReceivePipelineService

When the ReceivePipelineService executes, the following steps are performed:

Receives a message.
Uses the resolution framework to resolve the receive pipeline type.
Executes the receive pipeline.
[Optional] If the pipeline didn't set these properties, assigns BatchId and SequenceId promoted properties to the pipeline output message(s).
[Optional] If the resolution (resolutionDictionary) included a new itinerary for the pipeline output message, this new itinerary is applied to the pipeline output message(s), and the original itinerary is reserved for further processing of the original (inbound) message. If no itinerary was specified during the resolution, it is assumed that the pipeline output message(s) should continue the itinerary of the original message, and processing of the original message is discontinued.
Publishes the pipeline output messages.
[Optional] If a new itinerary was specified for the pipeline output message(s) during resolution, advances the itinerary of the original message and publishes the original inbound message (if it had outstanding itinerary steps).

The following drawing depicts an itinerary, which I have titled "OneWay-Debatch-MessageSendPort" (ODM) that uses the ReceivePipelineService:

Fairly simple scenario, you send an order batch through an itinerary onramp to BizTalk, where the ReceivePipelineService orchestration will pick it up, debatch it using the receive pipeline, and forwards the order items to the Message SendPort. (i.e. bound to a FILE location).

Debatching is one obvious application for the ReceivePipelineService, but you can specify any receive pipeline type in the resolver, so it can also be a 1:1 scenario where the pipeline has only 1 output message. I'm fairly certain that it will not work with FFDisAsm pipelines, though!

The SendPipelineService

When the SendPipelineService executes, the following steps are performed:

Receives the 1^st message within the batch of messages to be executed by the send pipeline. After receipt of this 1st message, a convoy subscription is created based on the Itinerary (should point to SendPipelineService) and the BatchId.
Resolves the SendPipelineServiceResolution object, necessary to perform the execution of the send pipeline. This resolution object contains the following parameters:
- Send pipeline type: the type of the send pipeline to execute.
- XLANGMessageComparer type: The SendPipelineService uses a serializable list to collect the messages to be aggregated. You can specify the comparer type to control how the messages are sorted. This happens in memory, but does not add much overhead compared to the use of Microsoft.XLANGs.Pipeline.SendPipelineInputMessages. I used an XLANGMessage comparer that compares the messages's SequenceId promoted property values.
- Batch Timeout: The timeout to apply to the entire batch. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- Message Timeout: The timeout to apply to each message. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
[Optional] If a new itinerary was specified during resolution, the new itinerary will be applied to the pipeline output message, and the original inbound message will continue it's own itinerary. If no new itinerary was specified, it is assumed that the original inbound message's itinerary should be applied to the aggregated message. So, if a new itinerary was specified, the original message's itinerary is advanced and applied to the original message, and the original message is republished for further processing.
The inbound message is added to a list of pipeline input messages.
The receive loop (batch convoy) starts.
The next message within the batch is received and added to the list of pipeline input messages.
[Optional] Like in step 3, if necessary, the original message's itinerary is advanced and the message republished.
When all messages have been received (or when a Batch or Message Timeout has occurred), the receive loop (and the corresponding batch convoy) is ended.
The pipeline input messages are sorted using the XLANGMessageComparer type, specified in the resolver.
The itinerary (new or original advanced) is applied to the pipeline output message, and the message is published.

Below a scenario that uses both the ReceivePipelineService and the SendPipelineService:

An orderbatch gets sent through an itinerary onramp, gets debatched by the ReceivePipelineService. The resolver does not specify a new itinerary, so the original message's itinerary is used for the debatched messages. The debatched messages all have the same BatchId, and have a current itinerary step that points to the SendPipelineService, so they get picked up by the SendPipelineService. The SendPipelineService sorts and aggregates the messages, and publishes one output message, which is sent to the offramp.

The Resequencer and ResequencerGo services

The idea for these services actually came from a TechEd (2006?) session, where Lee Graber detailed the use of the (what he called) Go-pattern to implement a resequencer. The Resequencer and ResequencerGo services work together to resequence any given sequence of messages. The ResequencerService blocks processing of a certain message until it receives the corresponding Go message (by using a parallel convoy). The ResequencerGoService's responsibility is to work out which message within the sequence is up next and publishes the corresponding Go message.

The ESB.Extension implementation let's the resolver decide which message is up next, so the resolver (i.e. BRE) could access a database or any other (in-memory?) representation of the message sequence (or batch) to determine who's next. In my BizUnit tests I have used the simplest possible scenario, where the maximum number of messages is always 5 and the next SequenceId is simple determined by incrementing by 1.

Below a scenario that uses the debatcher, aggregator and resequencer services:

In this scenario, the orderbatch gets debatched, every order item gets picked up by a seperate Sequencer instance. The 1st Go message, corresponding to the 1st message (SequenceId == 0) is sent to BizTalk, triggering the 1st Resequencer to release the 1st message. That message gets picked up by the SendPipelineService, and the SendPipelineService stores the pipeline input message, then itinerary-advances and republishes the original message, which triggers the ResequencerGoService to increment to the next SequenceId and publish the corresponding Go message. This process repeates until all messages have been processed (the ResequencerGo's resolution works out when it is the last message and does not publish a new Go message when it is the last). The SendPipelineService aggregates the message, applies the itinerary and ships it off to the offramp.

Conclusion

A select number of scenarios has been described here, and these are also the scenarios that have been tested. But there can be many many other scenarios, ranging from likely to not-in-a-million-years, that could be implemented with these services:

Receiving incoming messages for a certain interval (24 hours?), then sort and aggregate them and sending them off.
Debatching, sending the seperate items to web services, aggregating the responses, sending them back to the caller. (Doesn't work yet because Request-Response hasn't been implemented yet; future version)
Receiving incoming messages, block them with the ResequencerService, after a specified interval, start sending the messages to another system in a specified order.
If I missed an obvious scenario, please drop a comment!

Thanks,

Bram Veldhoen.

My beef with the ESB Toolkit 2.0 resolution framework

Bram Veldhoen — Fri, 28 Aug 2009 12:32:00 GMT

Lately, I have been working on an extension for the ESB Toolkit 2.0 for BizTalk Server 2009. It can be found here on CodePlex. More posts about this ESB.Extensions solution will follow later, this particular post will describe why I think the ESB Toolkit's resolution framework should be improved.

A short summary of how the ESB Toolkit resolution framework works. Resolvers are configured in the esb.config file using a specific name that can be coupled to a .NET type that needs to implement IResolveProvider. From the esb.config:

The above type implements IResolveProvider, and the ESB Toolkit provides a ResolverMgr and ResolverConfigHelper to allow for retrieving and caching the appropriate resolver types when the resolvers are retrieved from the itinerary and to be used by the itinerary service.

Let's look at IResolveProvider:

public interface IResolveProvider
{
    Dictionary<string, string> Resolve(ResolverInfo resolverInfo, XLANGMessage message);
    Dictionary<string, string> Resolve(string config, string resolver, XmlDocument message);
    Dictionary<string, string> Resolve(string config, string resolver, IBaseMessage message, IPipelineContext pipelineContext);
}

IResolveProvider provides three methods that all perform more or less the same task: resolve some information into a Dictionary<string, string>. My main problem with this approach it that everything that needs to be resolved needs to be in the form of string in order to be applicable for this way of resolution. I would expect a generic resolution framework to be more generic than that!

To make matters worse, there are limitations to which keys you are allowed to use in the resolution dictionary, if you're using the out of the box resolvers (i.e. the BRE resolver). Basically, you're stuck with the keys that are provided on the Microsoft.Practices.ESB.Resolver.Resolution type:

[Serializable]
public sealed class Resolution : IContextParameters, ITransformParameters, IEndpointParameters
{
... private members & constructor (left out for brevity) ...

    public string Action { get; set; }
    public string ActionField { get; set; }
    public string CacheTimeout { get; set; }
    public string DocumentSpecNameField { get; set; }
    public string DocumentSpecStrongNameField { get; set; }
    public string EndpointConfig { get; set; }
    public string EpmRRCorrelationTokenField { get; set; }
    public bool FixJaxRpc { get; set; }
    public string InboundTransportLocationField { get; set; }
    public string InboundTransportTypeField { get; set; }
    public string InterchangeIdField { get; set; }
    public string IsRequestResponseField { get; set; }
    public string MessageExchangePattern { get; set; }
    public string MessageType { get; set; }
    public string MethodNameField { get; set; }
    public string OutboundTransportCLSID { get; set; }
    public string ReceiveLocationNameField { get; set; }
    public string ReceivePortNameField { get; set; }
    public bool Success { get; set; }
    public string TargetNamespace { get; set; }
    public string TransformType { get; set; }
    public string TransportLocation { get; set; }
    public string TransportNamespace { get; set; }
    public string TransportType { get; set; }
    public string WindowUserField { get; set; }
}

Where the three Interfaces (IContextParameters, ITransformParameters, IEndpointParameters) all define a seperate set of properties (which are implemented by the Resolution type, see above).

The resolvers use this Resolution type to fill the (more generic) Dictionary<string, string> type using ResolverMgr.SetResolverDictionary:

public static void SetResolverDictionary(Resolution resolution, Dictionary<string, string> ResolverDictionary)
{
    try
    {
        ResolverDictionary.Add("Resolver.Action", resolution.Action ?? string.Empty);
        ResolverDictionary.Add("Resolver.ActionField", resolution.ActionField ?? string.Empty);

... more calls to ResolverDictionary.Add("Resolver.Xxx", resolution.Xxx ?? string.Empty); (removed for brevity) ...

        ResolverDictionary.Add("Resolver.TransportType", resolution.TransportType ?? string.Empty);
        ResolverDictionary.Add("Resolver.WindowUserField", resolution.WindowUserField ?? string.Empty);
    }

... some exception handling ...

}

This basically means that you can only work with the properties that the Resolution type provides. I see that as another limitation, because it means that the current resolvers are limited to only be able to resolve those properties. If I want to resolve a string value, that does not correspond to one of the properties of the Resolution type, I have two options:

Abuse one of the properties of the Resolution type. I.e. put a pipeline type name in the property TransformType. I started with this workaround, but it became a problem because I needed to resolve more complex objects than just strings (more on that later).
Write a custom Resolver. For instance, this means that I would need to implement a custom BRE resolver, that does resolution using the BRE in exactly the same way as the standard BRE resolver, just because of the different property.

The conclusion is that at the moment, the resolvers are coupled to what those resolvers are actually resolving. That should be decoupled if you ask me.

I took a stab at improving this situation by implementing a resolution framework that uses the following interface:

    public interface IResolutionProvider
    {
        ResolutionDictionary Resolve(ResolverInfo resolverInfo, XLANGMessage message);
        ResolutionDictionary Resolve(string config, string resolver, XmlDocument message);
        ResolutionDictionary Resolve(string config, string resolver, IBaseMessage message, IPipelineContext pipelineContext);
    }

Where ResolutionDictionary looks like this:

    [Serializable]
    public class ResolutionDictionary : Dictionary<string, object>, ISerializable
    {
        public ResolutionDictionary()
        {
        }

        public void SetValue(string key, object value)
        {
            base.Add(key, value);
        }

        public object GetValue(string key)
        {
            if (base.ContainsKey(key))
            {
                return base[key];
            }
            return null;
        }

        public string GetString(string key)
        {
            return GetValue(key) as string;
        }
    }

This dictionary allows resolvers to resolve anything, for as long as the resolution objects are serializable. It also allows for using any key value (string). The code, that's calling this kind of resolution, will be responsible for casting the object to the type that it expects.

The ESB.Extensions resolution framework, that uses the above IResolutionProvider and ResolutionDictionary, is still compatible with the esb configuration and the itinerary designer.

A full example can be found the in the ESB.Extensions solution on codeplex. I use the resolution framework there to resolve parameters needed in a couple of services: ReceivePipelineService, SendPipelineService and Resequencer services. I implemented some BRE vocabularies to enable for creating the resolution objects.

Off course, any feedback is welcome!

Promoted property bug in XmlDisAsm component?

Bram Veldhoen — Mon, 27 Jul 2009 19:06:15 GMT

I have just uploaded a bug report (#477852 on Microsoft Connect) regarding a highly specific scenario using the XmlReceive pipeline. It involves having an envelope and a document schema, in which the document schema contains an element (or attribute) that is promoted both as a distinguished AND as a promoted property. In this case, after processing by the XmlReceive pipeline, the property is only present in the message context as a distinguished property, and NOT as a promoted property. Some other people may have experienced this same issue:

(Trust me, it took me time to figure this one out).

I reproduced this behaviour both on a BizTalk Server 2009 deployment and in standalone mode with the ExecuteRecievePipelineStep, found in BizUnit 3.1. See the screenshots below.

(For the BizUnit ExecuteReceivePipelineStep intimi: I used test T_015, added both a distinguished and a promoted property to the Child1Attribute1 attribute (which is of type string) from schema0.xsd, and pointed it to promoted Property2 (which is of type string); then I ran the test. The screenshots above show that the promoted Property2 property isn't there).

I have not tested this with the FFDisAsm, I'll leave that to other enthousiasts.

Was it fun? Not always :). Was it useful? You decide!

BizTalk ESB Guidance 2.0 - Fixing the Itinerary OnRamp?

Bram Veldhoen — Tue, 24 Feb 2009 16:55:44 GMT

First off, for a very good presentation about what an ESB could be, watch Mark Richards here.

Recently, I have taken a look at the new-and-improved ESB Guidance 2.0 for BizTalk Server 2009 Beta. The ESB Guidance contains some interesting and potentially powerful concepts. Where BizTalk provides ESB-functionality out of the box (routing, transformation, etc., etc.), the ESB Guidance allows for a much more dynamic approach to realize these functionalities. With ESB Guidance, the majorities of these functionalities (i.e. dynamic routing and/or dynamic transformation) can be used by the client applications of the ESB (the systems sending messages to BizTalk) in the form of an ESB Guidance specific soap header called an Itinerary header. These Itinerary headers describe the processing steps that the message should go through within the ESB, and this Itinerary header could be different per message if that were necessary.

This approach offers a great deal of flexibility, because per each separate message coming from one of the ESB clients, it's path through the ESB can be set individually. However, it also means that those clients will need to be aware of the ESB Guidance Itinerary concept and have some way to add the Itinerary headers in the soap header of the message. Next to that, if there are changes to or within the ESB (configuration changes, system replacements), there's a chance that the Itinerary header will have to be changed as well.

In order to circumvent introducing this dependency of the ESB clients to the Itinerary concept, wouldn't it be a good idea to have the ESB be responsible for assigning the Itinerary headers? The most obvious place I can think of doing just that would be in a custom pipeline component, which can be configured with the appropriate Itinerary information, and which will just add this Itinerary header to the message as it comes in into the ESB. Taking this approach, the ContextAdder pipeline component, written by Jon Flanders, comes to mind. Using the ContextAdder, together with Saravana Kumar's article on design-time properties for custom pipeline components, I came up with a sample that does as I described above. You can download it here. BTW, the pipeline component in this sample writes or promotes any set of properties you want it to, it's not specific to Itineraries.

The sample needs BizTalk 2009 and ESB Guidance 2.0. I worked with the ESB Guidance 2.0 October 2008 release, so there may be some discrepancies when using it with the January 2009 release. Configuring the pipeline component is somewhat tedious, as the xml syntax needs to be very specific (especially regarding &lt;, < and <). If using it for real, a way to programmatically generate the config would be a good enhancement.

The sample uses FILE receive locations only and uses a very simple Itinerary that uses the orchestration routing and transformation services, configured with the STATIC resolvers. A screenshot of the pipeline configuration is displayed below.

I'd be very interested in other's thoughts on this approach, so drop me a comment!

[27-02-2009] Nick Hauenstein pointed out that the January 2009 CTP2 release of ESB Guidance 2.0 contains a much better way to achieve the same result: static Itineraries, which can be set within a pipeline component. Check out his post!

Combining BizTalk Software Factory and BizTalk Deployment Framework

Bram Veldhoen — Wed, 22 Oct 2008 18:07:12 GMT

Recently I have finished a preliminary version of an adaptation of the BizTalk Software Factory guidance automation package, owned by Jean-Paul Smit and Dick Dijkstra (BSF), where I have extended the functionality of the BizTalk Software Factory with support for Scott Colestock's BizTalk Deployment Framework (BDF). The latest changeset (17321) of the BSF source contains these changes, so you can testdrive it yourself!

The main advantage of using BSF and BDF in conjunction is that you should be able to produce a full BizTalk compilable and deployable solution within minutes using a few mouseclicks (ok, and some keyboard entries), leaving a situation where you have a repeatable development and deployment process from day 1, whilst maintaining strict naming conventions. After generating the solution, you will still be able to enjoy the full functionality of the BDF.

During the wizard pages of the BSF, you will be prompted to enter some BDF specific settings:

PreProcess Bindings: If unchecked, you will get a regular bindings file that will not be processed by ElementTunnel.exe or XmlPreProcess.exe. If checked, you will get a bindingsMaster file, that will be processed by the two mentioned exe's.

Add VDirList: If unchecked, you will not get a VDirList file (so you won't be exposing webservices). If checked, you will get a VDirList file, that you will have to edit manually to contain your vdir mappings.

Company Name: The company name that is used in the generation of the Windows Installer (msi).

Copy DeployTools, Copy WiXSetup: If unchecked, no files will be copied. If checked, BSF will copy the BDF DeployTools and WiXSetup directories into your solution root folder. To keep things simple, the files in these directories will not be added to your solution.

The BDF offers many many more settings to play with, but I decided to keep things simple and limit the configurable settings to those that would alter the solution structure or the files within the solution.

The BSF will generate the following solution artifacts:

So after preparing your environment for BDF, you should be able to deploy your (completely empty) solution to BizTalk via the generated msi! (To do this: select the *.WiXSetup.build file in solution explorer, then in VS2005 Tools execute "NAnt Current Target").

To prepare your environment, you basically only have to run MakeExternalTools_VS2005.vbs and make sure all paths in this .vbs file are correct (see BizTalkSoftwareFactory\BDF\Info), but refer to the BDF docs for complete documentation.

Have fun.

Unit testing BizTalk artifacts using BizUnit

Bram Veldhoen — Wed, 22 Oct 2008 17:30:40 GMT

First off, credit where credit's due: Tomas Restrepo has done an amazing job making pipeline testing way easier with the pipeline testing library. Read all about it here.

Tomas' library inspired me to write some test steps for the BizUnit framework that would allow for true unit testing of BizTalk components: being able to test BizTalk components without having to deploy the components to a BizTalk server, making testing roundtrips easier and overall less painfull. As additional advantage, you can have all tests use a similar approach/syntax, and use the already present validation steps to validate the results. I have implemented the following steps for BizUnit:
- ExecuteMapStep
- ExecuteReceivePipelineStep
- ExecuteSendPipelineStep

Obviously, the latter two are using Tomas' pipeline testing library.

The ExecuteMapStep step can be used to test biztalk maps, using an input xml file. An example of a BizUnit test, that executes a map and validates the resulting output xml file (via the BizUnit context), is displayed below.

<TestCase testName="T_006_ExecuteMapStepWithValidationWithContextLoader">
  <TestExecution>
    <TestStep assemblyPath="BizUnit.BizTalkSteps.dll" typeName="BizUnit.BizTalkSteps.ExecuteMapStep">
      <Map assemblyPath="..\..\..\Test\BizUnit.BizTalkTestArtifacts\bin\Development\BizUnit.BizTalkTestArtifacts.dll" typeName="BizUnit.BizTalkTestArtifacts.MapSchema1ToSchema2"/>
      <Source>..\..\..\Test\BizUnit.BizTalkTestArtifacts\Instances\Schema1.xml</Source>
      <Destination>Schema2.006.xml</Destination>
      <ContextLoaderStep assemblyPath="" typeName="BizUnit.XmlContextLoader">
        <XPath contextKey="testValue">/*[local-name()='Schema2Root' and namespace-uri()='http://BizUnit.BizTalkTestArtifacts.Schema2']/*[local-name()='Child1' and namespace-uri()='']/@*[local-name()='Child1Attribute1' and namespace-uri()='']</XPath>
      </ContextLoaderStep>
      <ValidationStep assemblyPath="" typeName="BizUnit.ContextValidationStep">
        <Context keyName="testValue">1</Context>
      </ValidationStep>
    </TestStep>
  </TestExecution>
</TestCase>

The ExecuteReceivePipelineStep and ExecuteSendPipelineStep let you test receive or send pipelines. The input for both the ExecuteReceivePipelineStep and ExecuteSendPipelineStep (optionally) consists of:
- Per-instance pipeline configuration
- Document specs
- Input xml file (multiple input files can be specified for the ExecuteSendPipelineStep)
- Message context input file(s): an xml file, containing all context properties (name, namespace, value, and whether these properties are promoted or not)

ExecuteReceivePipelineStep:

<TestStep assemblyPath="BizUnit.BizTalkSteps.dll" typeName="BizUnit.BizTalkSteps.ExecuteReceivePipelineStep">
  <Pipeline assemblyPath="..\..\..\Test\BizUnit.BizTalkTestArtifacts\bin\Development\BizUnit.BizTalkTestArtifacts.dll" typeName="BizUnit.BizTalkTestArtifacts.ReceivePipeline1">
    <InstanceConfigFile>..\..\..\Test\BizUnit.BizTalkSteps.Tests\TestData\ReceivePipeline3.InstanceConfig.xml</InstanceConfigFile>
    <DocSpecs>
      <DocSpec assemblyPath="..\..\..\Test\BizUnit.BizTalkTestArtifacts\bin\Development\BizUnit.BizTalkTestArtifacts.dll" typeName="BizUnit.BizTalkTestArtifacts.Schema0" />
      <DocSpec assemblyPath="..\..\..\Test\BizUnit.BizTalkTestArtifacts\bin\Development\BizUnit.BizTalkTestArtifacts.dll" typeName="BizUnit.BizTalkTestArtifacts.Schema3Env" />
    </DocSpecs>
  </Pipeline>
  <InputContextFile>..\..\..\Test\BizUnit.BizTalkSteps.Tests\TestData\Context.0.xml</InputContextFile>
  <Source>..\..\..\Test\BizUnit.BizTalkTestArtifacts\Instances\Schema3Env.xml</Source>
  <DestinationFileFormat>Output015.{0}.xml</DestinationFileFormat>
  <OutputContextFileFormat>Context015.{0}.xml</OutputContextFileFormat>
</TestStep>

ExecuteSendPipelineStep:

<TestStep assemblyPath="BizUnit.BizTalkSteps.dll" typeName="BizUnit.BizTalkSteps.ExecuteSendPipelineStep">
  <Pipeline assemblyPath="..\..\..\Test\BizUnit.BizTalkTestArtifacts\bin\Development\BizUnit.BizTalkTestArtifacts.dll" typeName="BizUnit.BizTalkTestArtifacts.SendPipeline1">
    <InstanceConfigFile>..\..\..\Test\BizUnit.BizTalkSteps.Tests\TestData\SendPipeline3.InstanceConfig.xml</InstanceConfigFile>
    <DocSpecs>
      <DocSpec assemblyPath="..\..\..\Test\BizUnit.BizTalkTestArtifacts\bin\Development\BizUnit.BizTalkTestArtifacts.dll" typeName="BizUnit.BizTalkTestArtifacts.Schema0" />
      <DocSpec assemblyPath="..\..\..\Test\BizUnit.BizTalkTestArtifacts\bin\Development\BizUnit.BizTalkTestArtifacts.dll" typeName="BizUnit.BizTalkTestArtifacts.Schema3Env" />
    </DocSpecs>
  </Pipeline>
  <InputContextDir>..\..\..\Test\BizUnit.BizTalkSteps.Tests\TestData</InputContextDir>
  <InputContextSearchPattern>Context*.xml</InputContextSearchPattern>
  <SourceDir>..\..\..\Test\BizUnit.BizTalkTestArtifacts\Instances\</SourceDir>
  <SearchPattern>Child*.xml</SearchPattern>
  <Destination>Output.025.xml</Destination>
  <OutputContextFile>Context.024.xml</OutputContextFile>
</TestStep>

Per-instance pipeline configuration:

Below the per-instance pipeline configuration of a pipeline, containing only the Xml Disassembler pipeline component.

<?xml version="1.0" encoding="utf-8"?>
<Root xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Stages>
    <Stage CategoryId="9d0e4105-4cce-4536-83fa-4a5040674ad6">
      <Components>
        <Component Name="Microsoft.BizTalk.Component.XmlDasmComp">
          <Properties>
            <EnvelopeSpecNames vt="8">BizUnit.BizTalkTestArtifacts.Schema3Env</EnvelopeSpecNames>
            <EnvelopeSpecTargetNamespaces vt="8">http://BizUnit.BizTalkTestArtifacts.Schema3Env</EnvelopeSpecTargetNamespaces>
            <DocumentSpecNames vt="8">BizUnit.BizTalkTestArtifacts.Schema0</DocumentSpecNames>
            <DocumentSpecTargetNamespaces vt="8">http://BizUnit.BizTalkTestArtifacts.Schema0</DocumentSpecTargetNamespaces>
            <AllowUnrecognizedMessage vt="11">false</AllowUnrecognizedMessage>
            <ValidateDocument vt="11">false</ValidateDocument>
            <RecoverableInterchangeProcessing vt="11">false</RecoverableInterchangeProcessing>
            <HiddenProperties vt="8">EnvelopeSpecTargetNamespaces,DocumentSpecTargetNamespaces</HiddenProperties>
          </Properties>
        </Component>
      </Components>
    </Stage>
  </Stages>
</Root>

Context xml file:

<MessageInfo xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <ContextInfo PropertiesCount="1">
    <Property Name="Property1" Namespace="http://BizUnit.BizTalkTestArtifacts.PropertySchema1" Value="10" Promoted="true" />
  </ContextInfo>
</MessageInfo>

The code (including tests) have already been uploaded to the owner of BizUnit. I don't know when the next release for BizUnit is planned. If you need the code for these BizUnit steps, send me an email.

The output of the steps consist of:
- Xml output file
- Message context xml file, with the same format as the input context xml file..

The example below displays a BizUnit validation step snippet that checks if a certain property was promoted and has value 10:

[19-07-2009]: The ExecuteMapStep, ExecuteReceivePipelineStep and ExecuteSendPipelineStep have been included in BizUnit 3.1, which is now available at codeplex. Thanks, Kevin!

Messaging-only request-response correlation

Bram Veldhoen — Wed, 20 Aug 2008 21:21:36 GMT

Recently I was asked if it was possible to "couple" two services via BizTalk without using orchestrations. Let's look at that required scenario more closely:

1. A request message (ReqA) is received on the WCF-BasicHttp Receive Adapter, and it is posted to the MsgBox.
2. A solicit-response send port is activated due to a subscribtion to ReqA.
3. ReqA is mapped to ReqB.
4. ReqB is sent via the WCF-BasicHttp Send Adapter
5. RespB is received by the WCF-BasicHttp Send Adapter.
6. RespB is mapped to RespA and it is posted to the MsgBox.
6. RespB is picked up by the appropriate instance of the request-response receive port due to an instance subscription.
7. RespA is sent via the WCF-BasicHttp Receive Adapter.

The first questions that came up with me were:
1. How can the response of the outgoing webservice call be correlated to the incoming one? In the above scenario, that question would be "How can response B be correlated to response A?". I certainly didn't know that that comes out of the box with BizTalk. How? We'll see that later on.
2. What happens if we configure multple (outgoing) solicit-response send ports, subscribing to messages coming in on the one request-response receive port? Will timing issues start to play a role? Or will an exception be thrown?

Before answering these questions, I would like to say that SoapUI is simply AWESOME when developing/working with web services (yes, including WCF web services). True, it's JAVA application, but still... :) It blows other webservice related stuff I've seen out of the water. Give it a try! In the above example, I used SoapUI to submit requests/receive responses to/from the WCF service, coupled to the request-response receive location.

In order to answer question 1, we could check the instance subscription generated when a message is received on the request-response receive port (by stopping -but not unenlisting- the subscribing sendport and submitting a request), and it's as follows:

http://schemas.microsoft.com/BizTalk/2003/system-properties.EpmRRCorrelationToken == Svr:WIN2003EER2SP2;PID:3092;RRUid:{9731CF80-DDCF-4BB8-9CCA-03651243AE3E} And
http://schemas.microsoft.com/BizTalk/2003/system-properties.RouteDirectToTP == True

Documentation on those properties is hard to come by. The BizTalk documentation states:

BTS.EpmRRCorrelationToken
Promoted by the Messaging Engine on request-response message execution. The property is promoted before messages are submitted into the MessageBox database.
Used internally by the Messaging Engine. Specifies the Server Name, Process ID and a unique GUID for a request response stream of messages.

BTS.RouteDirectToTP
Promoted by the Messaging Engine on messages for loop back or request-response execution. The property is promoted before messages are submitted into the MessageBox database.
Used internally by the Messaging Engine to enable loop back and request-response scenarios.

Anyone will agree that <censored>. However, from the fact that the above example is indeed working, we can assume that the EpmRRCorrelationToken property value is copied to the contexts of all messages involved in the communication flow. This implies that the EpmRRCorrelationToken property "survives" maps, the WCF solicit-response adapter and the XmlReceive pipeline. If you'd check the message context of the response message, published by the solicit-response port, you'll find those properties present and promoted.

Question 2 can be answered fairly quickly. Configure & start a second solicit-response send port, and then fire a message into the request-response receive port (i.e. using SoapUI). This will give you a routing failure in BizTalk and the following soap fault on the soap client:

"The message found multiple request response subscriptions. A message can only be routed to a single request response subscription."

Which basically means that a message just won't get processed in this scenario. Makes sense, but would you have predicted that? Searching the web found a hotfix (KB923632), that apparently enables scenarios where multiple request response subscriptions should be allowed. If you can think of a decent scenario where this would be needed, leave a comment!

How could we use this information?
One scenario could be to have an orchestration that uses two separate direct bound orchestration ports (one receive and one send), instead of one receiving request-response orchestration port to process the request and to submit a response. So instead of having an original request-response port, bound to a physical request-response port ("specify later"), like this:

...we create an orchestration with two separate direct bound ports like this:

Where the receive shape (Receive_1) has the following filter expression: (BTS.MessageType == "http://Blah.BlahBlah.Schema1#Schema1Root")
The SetContext expression shape looked like this: Message_2(*) = Message_1(*);

The result was that the response message (actually, better to say the transformed message, because we're intentionally trying to avoid request-response terminology here) couldn't be routed back to the physical request-response port (routing failure). Checking the message context revealed that the EpmRRCorrelationToken and RouteDirectToTP were present (due to copying over the properties), but they were not promoted, which is correct (because setting properties in an expression shape within an orchestration does not promote them). This can be solved by adding a correlation type and set, using these two context properties, and setting the send shape to initialize this correlation set. The result was that the message was correctly picked up by the subscribing request-response port, but another error occurred::

Event Type: Error
Event Source:   BizTalk Server 2006
Event Category: BizTalk Server 2006
Event ID:   5813
Date:       8/19/2008
Time:       11:17:27 PM
User:       N/A
Computer:   WIN2003EER2SP2
Description:
A response message for two-way receive port "ReceivePort1" is being suspended as the messaging engine could not correlate the response to an existing request message. This usually happens when the host process has been recycled.

MessageId: {A817F71C-5AC0-4CEB-BAAC-C60EFED8D92E}
InstanceID: {B931AE36-5C9F-4B23-8697-DCC79E0A76DF}

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Hmmmm, what's all this about? Apparently some required properties, that are used internally by BizTalk for request response correlation, weren't satisfied. After some fiddling around with context properties, performing a lot of WinMerge actions to compare message contexts (of the messages coming from the solicit-response ports (see above) and the messages coming from the orchestration), and most importantly, by finding this excellent post by Paolo Salvatori, describing exactly the property set needed, I changed the expression shape and the correlation type into the following:

Expression shape:
Message_2(*) = Message_1(*);
Message_2(BTS.RouteDirectToTP) = true;
Message_2(BTS.IsRequestResponse) = true;

Correlation Type:
BTS.CorrelationToken,BTS.EpmRRCorrelationToken,BTS.IsRequestResponse,BTS.ReqRespTransmitPipelineID,BTS.RouteDirectToTP

The result was that the response was correctly picked up & processed by the instance-subscribing physical request-response port.

Conclusion
A much overlooked feature of BizTalk is the ability to route responses without the need for orchestrations by using messaging only. This post described what happens internally in such a scenario and also described how that information can be used to create an orchestration, in which the request and response can be communicated over separate direct bound orchestration ports instead of over one request-response port, which can be helpful (for instance) for testing the orchestration using physical FILE ports.

Post-receive debatching without orchestrations

Bram Veldhoen — Tue, 12 Aug 2008 03:39:35 GMT

This post describes a way to perform debatching of a message after the original message has already been received and submitted to the BizTalk message box. This can be the case when for instance there are subscribers defined in BizTalk that subscribe to:
- The original message before debatching. To my knowledge, it isn't possible to post both the original message and the debatched messages to the BizTalk MsgBox.
- A mapped version of the original message. To my knowledge, it isn't possible to post both the original and the mapped messages to the BizTalk MsgBox.

So how do we handle a message in the BizTalk MsgBox, that needs to be debatched into seperate messages (without orchestrations)? Below a schematic representation of the solution used in this sample.

org = Original Message: contains some elements that occur 0..n.
arr = Array Message: a non-envelope message containing 0..n entities to debatch
env = Envelope Message: BizTalk envelope message with identical structure to array message
msg 1, msg 2, ..., msg n = Debatched Messages

The steps in the above scenario:
1. A solicit-response send port has a subscription filter for the original message.
2. An outbound map is performed, transforming the original message into an array message.
3. Within the NamespaceReplacing pipeline component, the namespace for the array message is replaced with the namespace of the envelope message (this is in fact a transformation, though not via a BizTalk map).
4. The Loopback Adapter bounces the envelope message.
5. The XmlDisassemble component in the XmlReceive pipeline debatches (splits) the envelope message into its seperate messages.
6. The seperate messages are posted to the BizTalk MsgBox for further processing.

In order to make this work, there are some specific requirements to how the BizTalk schemas are setup. Below the representation of the used BizTalk schemas.

org = Original Message = OrdersRequest
arr = Array Message = InternalOrders
env = Envelope Message = InternalOrdersEnvelope
msg 1, msg 2, ..., msg n = Debatched Messages = InternalOrder

Key thing here is that the array schema has an xml structure similar to the envelope schema. In fact, the envelope schema imports the array schema, and the envelope schema's root element type is set to be the same as the root element type of the array schema. The only differences between the array schema and the envelope schema are:
- The root namespace.
- The envelope schema is configured to be a BizTalk envelope schema, while the array schema is not.

So why the Namespace replacing?
You may have noticed that the above could also be done by mapping from the original message to the envelope message directly, and not (have to) use the namespace replacing component. However, if you will try this, you will find that BizTalk doesn't allow to map TO envelope schemas, when that map is configured as inbound or outbound map on a BizTalk physical port. You'll get an error like:

Event Type: Error
Event Source: BizTalk Server 2006
Event Category: BizTalk Server 2006
Event ID: 5782
Date:  9/5/2008
Time:  10:19:52 PM
User:  N/A
Computer: WINXP
Description:
The Messaging Engine failed while executing the outbound map for the message going to the destination URL "LOOPBACK://cfc0607f-e119-4e6b-ae34-8fae2a92b084" with the Message Type "http://Samples.Messaging.Schemas.OrdersRequest#OrdersRequest". Details:"Document type "http://Samples.Messaging.Schemas.InternalOrder#InternalOrder" does not match any of the given schemas. "

I will send a chocky bar of choice to the first one to find the solution for this error, whereever you are!

In order to work around this issue, I have used the Namespace replacer component to perform a 1:1 map from the array schema to the envelope schema by replacing the root namespace. A true BizTalk map, with all elements from the array schema 'connected' to the corresponding elements in the envelope schema, should have had the same effect (were it not for the fact that BizTalk doesn't allow that on ports).
The Namespace replacer component has been implemented by using a lot of code from the Microsoft.Practices.ESB Namespace pipeline components (RemoveNamespace component). The Namespace replacer component takes two configuration parameters: OldNamespace and NewNamespace, and it just substitues the OldNamespace on all elements and attributes it encounters with the NewNamespace.

The XmlReceive pipeline configuration in this sample is pretty standard: debatching using an envelope schema, exactly like you would do on a 'normal' receive port.

This sample provides a way to debatch (or normalize) a message on a single port without the need for orchestrations. Think of it as a debatching port or normalizing port. There are some limitations to this approach:
- Error handling isn't possible on a per-debatched-message basis. If debatching fails (on one debatched message), I assume you will get one suspended envelope message. When using orchestrations for debatching, the error handling could be done per debatched message. [Update] This could (to some extent) be mitigated by turning on Recoverable Interchange Processing on the XmlReceive Pipeline in combination with Failed Message Routing.
- ...? Any additions are welcome.

The sample runs on BizTalk Server 2006 R2. You can download it. N.B. Updated on 05-09-2008: The "Copy properties" property of the LOOPBACK adapter needs to be set to False for this sample to work.

Some other techniques for debatching messages are described here. You can find the Loopback adapter here.

UPDATE (additional info):

Recently, I was pointed out that a more common way to re-use xsd types is by using element refs. I have tested this sample to see if it still works when using element refs instead of re-using the xsd types, and I found that it still works. Here's the changed sample using element ref in the envelope schema.

Update [05-10-2008]:

If you're interested in the namespace replacement, you're much better off writing a custom pipeline component that uses the Microsoft.BizTalk.Component.NamespaceTranslatorStream class, found in Microsoft.BizTalk.Pipeline.Components.

Recursive Orchestration

Bram Veldhoen — Wed, 23 Jul 2008 05:44:49 GMT

This post describes how to implement a recursive BizTalk orchestration using correlation sets. Using a recursive orchestration came up as a possible solution during one of my previous projects, where a message needed to be resubmitted with a changed status code, so that a new instance of the same orchestration type could pick it up for processing. Next to that, it's a fun excersize!

You can't recurse orchestrations using the Call Orchestration or Start Orchestration shapes, because the Orchestration designer just doesn't let you select the same orchestration type as the one you're trying to start/call from. I haven't tried fooling the orchestration editor by editing the .odx file in the xml editor, but I can't imagine any other result than a compile-time error. From the BizTalk Server 2006 R2 Documentation:

Caution

BizTalk Server 2006 does not support recursive orchestrations. If Orchestration A calls or starts Orchestration B, then Orchestration B cannot call or start Orchestration A directly, nor can it call or start any orchestration that directly or indirectly calls Orchestration A.

So assuming we have to rule out the use of the Call/Start orchestration shapes, we have to use another way to do the correlation. In come the correlation sets. The BizTalk documentation uses orchestration A and B in examples for orchestration communication (as in the example above), often assuming that orchestrations A and B are of different types. In the example below, I have used a standard way of correlation where orchestration A’s type equals orchestration B’s type.

It breaks down into 3 parts, where each part has two branches. The first part is where the request message gets received, the second part is where the child orchestration gets ‘called’ (using direct bound ports and correlation sets), and the third part is where the response gets sent. The correlation type used consists of only a guid (ctInstandeId), and the orchestration uses two correlation sets, both based on that one correlation type (csInstanceId1, csInstanceId2).

The key point is to understand what using correlation sets does to the message context and subscriptions. The BizTalk documentation is more accurate (ms-help://MS.BTS.2006/BTS06CoreDocs/html/528dcd6c-d364-4bb8-8deb-cd4a0791867f.htm), but in my own words:

An initializing receive creates ‘Exists’ subscription criteria on all correlation type properties. In the example orchestration, we create this ‘Exists’ subscription manually by using the filter expression on the RCVRECREQ receive shape.
An initializing send promotes the correlation type properties to the message context as promoted properties. In the example orchestration, we do this on the SNDRECREQ send shape.
A following receive creates an instance subscription based on the correlation type properties. In the example orchestration, we do this on the RCVRECRESP receive shape.
A following send promotes the correlation type properties to the message context as promoted properties. In the example orchestration, we don’t use this.

Subscriptions

This is what the subscriptions look like.

RCVRECREQ (activation subscription):

http://schemas.microsoft.com/BizTalk/2003/system-properties.MessageType == http://RecursiveOrch.Request#Request And

http://RecursiveOrch.Properties.InstanceId Exists

RCVRECRESP (instance subscription):

http://RecursiveOrch.Properties.InstanceId == c1c86b9c-4d01-4176-9a19-3908f46fe053 And

http://schemas.microsoft.com/BizTalk/2003/system-properties.MessageType == http://RecursiveOrch.Response#Response

DebugView Output

00000000                        0.00000000                    [5844] RCVINITREQ, id=1, count=0.
00000001                        0.00035004                    [5844] SNDRECREQ, id=1, count=1.
00000002                        0.24793179                    [5844] RCVRECREQ, id=1, count=1.
00000003                        0.25474858                    [5844] SNDRECREQ, id=1, count=2.
00000004                        0.75415730                    [5844] RCVRECREQ, id=1, count=2.
00000005                        0.75665259                    [5844] SNDRECREQ, id=1, count=3.
00000006                        1.26953459                    [5844] RCVRECREQ, id=1, count=3.
00000007                        1.27350295                    [5844] RecursiveOrch - Recursion terminated for request, id=1, count=3.
00000008                        1.92181325                    [5844] SNDRECRESP, id=1, count=3.
00000009                        2.32890010                    [5844] RCVRECRESP, id=1, count=3.
00000010                        2.33078957                    [5844] SNDRECRESP, id=1, count=3.
00000011                        2.53953433                    [5844] RCVRECRESP, id=1, count=3.
00000012                        2.54006386                    [5844] SNDRECRESP, id=1, count=3.
00000013                        3.03803825                    [5844] RCVRECRESP, id=1, count=3.
00000014                        3.04450655                    [5844] SNDFINALRESP, id=1, count=3.

Download

You can download the solution here.

WCF Latency Test Harness and WCF Performance

Bram Veldhoen — Sat, 26 Jan 2008 22:14:02 GMT

On performance measurements in general

Performance measurements are highly sensitive to details. One implementation detail can have a huge impact. The performance measurement results, posted here, apply to the implementation, that was used to generate the measurements. You can find the link to that implementation, the WCF Test Harness code (on codeplex), below.

The main reasons to blog about the performance measurement results is to receive feedback and validation of the results by others, who are also interested in it, and to verify that the implementation doesn't contain any issues that might invalidate the results. If you find anything wrong with the code, have trouble reproducing, etc., pls. leave a comment and I'll get back to it asap.

Throughput vs Latency

The implementation of the WCF Latency Test Harness is focussed on the latency of a range of methods, implemented by a WCF service and called by a WCF client. For throughput measurement on WCF services, pls. go to this Microsoft whitepaper.

Variations during the performance tests

The WCF Latency Test Harness allows to measure the latency of WCF whilst varying the following parameters:

Test scope: Direct, SameAppDomain, SameProcess, SameMachine, Remote
Bindings: nullTransport, netNamedPipe, netTcp, netMsmq, basicHttp, wsHttp (and .NET Remoting).
instancingModes: Single, PerSession, PerCall
Serializers: DataContractSerializer (DefaultSF (for Remoting, this is the BinaryFormatter)), PreserveObjectReferencesDataContractSerializer (PORDCSF), NetDataContractSerializer (NDCSF), XmlSerializer (XMLSF)
CallModes: OneWay, TwoWay
ReceiveDataMethods: several methods are implemented, each using a different data type for the data to be transferred, ranging from byte[], string, Stream to more complex data structures.
DataSizes: size of the used data type/data structure to be transferred.

You'll find enums for most of the parameters above in the implementation (if not already provided by the .NET Framework). Most of these parameters can be manipulated by changing the configuration only.

Overview of measurement method

Before the tests start, the appropriate configuration files are copied to the service and client private application paths. The service is self-hosted and loads the configuration from those copied files.

The implementation contains a WCF service, which exposes a number of methods via a ServiceContract and DataContract. It also contains a WCF client, which calls the WCF service methods within a loop.

The latency measurements of the WCF service methods are done using the System.Diagnostics.Stopwatch. The client call to the WCF service is synchronized (even in a OneWay scenario) by using a 2nd WCF netTcp service, hosted by the client, which receives the receipt signal of the service. There is never more than one call on the service at any one time. This also means that the queueing mechanism of WCF is not tested to the full extent. Since the aim of the WCF Latency Test Harness is to measure the latency, this is acceptable.

Data structures

In order to test methods with different parameters, the following data structures were used.

Member variables data types:

Data: byte[] of size DataSize.

Text: string of size DataSize.

AdvancedContainer.Containers: AdvancedContainer[].

AdvancedTextContainer.Containers: AdvancedTextContainer[].

In the tests with the AdvancedContainer and AdvancedTextContainer data structures, an object hierarchy of 1x3x3 was used (One parent, three children, with each three children), resulting in a memory footprint of about 10x the footprint of the SimpleContainer and SimpleTextContainer (for the same DataSize).

Customizing the WCF Latency Test Harness

Additional bindings can be tested by adding some configuration, including binding configuration. Testing the impact of configuration settings can be done by changing the binding configuration file. For other additions or changes (additional data structures, etc.), changes to the code are necessary.

Security

WCF is secure by default. In the WCF Latency Test Harness implementation, in the default configuration (RequestedBindingConfigTags=Default), all security has been disabled for the WCF bindings. The impact of enabled security can be tested by adjusting the configuration files and configuring certificates etc.

Test parameter combinations

Some combinations of the test parameters above just don't fly, or weren't implemented due to other reasons:

netMsmq and TwoWay. WCF enforces that methods are decorated with the OneWay=true OperationContractAttribute parameter in a netMsmq scenario.
netMsmq and PerSession.
.NET Remoting and other-than DefaultSF Serializer. .NET Remoting uses the BinaryFormatter, while the other WCF Bindings use the default DataContract serializer. Hence the term "DefaultSF" (Default Serializer Format).
nullTransport and other-than SameAppDomain test scope.

Target system specifications

The measurements were performed on the following system:

Intel Core 2 Duo E4500 2.20 GHz CPU
2.0 Gb RAM
32-bit Vista Business, .NET 3.5

Results

Results were generated using Changeset 5849 of the source code.

The raw data of the graphs, presented below, can be found in the file SameAppDomain.Single.zip here. This file also contains the configuration files, used to produce these results, and the performance counter log output (CPU usage etc.).

N.B. All graphs are with double logarithmic scales.

Time for some text here:

basicHttp and wsHttp performed equally. We haven't been using any of the ws specific settings, so that probably explains a lot :).
nullTransport, netNamedPipe and netMsmq seemed to converge for large DataSizes (10⁶+ bytes).

In the last graph, we see the effect that .NET Remoting outperforms netTcp using the default DataContractSerializer when using a more complex data structure. A detailed discussion and explanation can be found here.

Streaming and WCF TransferMode

In a TwoWay or buffered scenario, the difference between ReceiveData and ReceiveStream was negligible. In a OneWay Streamed scenario, using the transferMode=Streamed setting was beneficial at DataSizes > 10,000.

Performance Counters (CPU Usage)

The performance counters to be logged during the measurements with the WCF Latency Test Harness can be configured in one of the configuration files by adding the logman.exe/perfmon.exe compatible performance counter name.

The above graphs show that the CPU usage was lower than 100% at all times during the measurements, which indicates that the latency measurements were performed correctly regarding the total system load, as latency may degrade when the total CPU usage nears 100%. Such a scenario (near-100% CPU usage) would be appropriate for throughput measurements, but not for latency measurements).

Source code

http://www.codeplex.com/WCFLatencyTstHarness.

Further measurements

Other measurements, that were not performed, but seem interesting:

Other OS: XP, 64-bit Vista
Bindings configured to use transport and/or message security

Acknowledgements

The following people have contributed to the WCF Latency Test Harness:

Ilan Tavor (KLA-Tencor, making it possible)
Alon Fliess (technical guidance)
Sasha Goldshtein (technical input on the serialization aspects; Kudos!)
Josh Reuben (general technical input)
Manu Cohen-Yashar