BizTalk 2006 R2

Moving DataSets through BizTalk

EltonStoneman — Thu, 06 May 2010 12:25:31 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

Yuck. But sometimes you have to, so here are a couple of things to bear in mind:

Schemas

Point a codegen tool at a WCF endpoint which exposes a DataSet and it will generate an XSD which describes the DataSet like this:

<xs:elementminOccurs="0"name="GetDataSetResult"nillable="true">

<xs:complexType>

<xs:annotation>

<xs:appinfo>

<ActualTypeName="DataSet"

Namespace="http://schemas.datacontract.org/2004/07/System.Data"

xmlns="http://schemas.microsoft.com/2003/10/Serialization/" />

</xs:appinfo>

</xs:annotation>

<xs:sequence>

<xs:elementref="xs:schema" />

<xs:any />

</xs:sequence>

</xs:complexType>

</xs:element>

In a serialized instance, the element of type xs:schema contains a full schema which describes the structure of the DataSet – tables, columns etc. The second element, of type xs:any, contains the actual content of the DataSet, expressed as DiffGrams:

<xs:schemaid="NewDataSet"xmlns:xs="http://www.w3.org/2001/XMLSchema"xmlns=""xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">

<xs:elementname="NewDataSet"msdata:IsDataSet="true"msdata:UseCurrentLocale="true">

<xs:complexType>

<xs:choiceminOccurs="0"maxOccurs="unbounded">

<xs:elementname="Table1">

<xs:complexType>

<xs:sequence>

<xs:elementname="Id"type="xs:string"minOccurs="0" />

<xs:elementname="Name"type="xs:string"minOccurs="0" />

<xs:elementname="Date"type="xs:string"minOccurs="0" />

</xs:sequence>

</xs:complexType>

</xs:element>

</xs:choice>

</xs:complexType>

</xs:element>

</xs:schema>

<diffgr:diffgramxmlns:diffgr="urn:schemas-microsoft-com:xml-diffgram-v1"xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">

<NewDataSetxmlns="">

<Table1diffgr:id="Table11"msdata:rowOrder="0"diffgr:hasChanges="inserted">

</Table1>

</NewDataSet>

</diffgr:diffgram>

</GetDataSetResult>

Put the XSD into a BizTalk schema and it will fail to compile, giving you error: The 'http://www.w3.org/2001/XMLSchema:schema' element is not declared. You should be able to work around that, but I've had no luck in BizTalk Server 2006 R2 – instead you can safely change that xs:schema element to be another xs:any type:

<xs:elementminOccurs="0"name="GetDataSetResult"nillable="true">

<xs:complexType>

<xs:sequence>

<xs:any />

<xs:any />

</xs:sequence>

</xs:complexType>

</xs:element>

(This snippet omits the annotation, but you can leave it in the schema).

For an XML instance to pass validation through the schema, you'll also need to flag the any attributes so they can contain any namespace and skip validation:

<xs:elementminOccurs="0"name="GetDataSetResult"nillable="true">

<xs:complexType>

<xs:sequence>

<xs:anynamespace="##any"processContents="skip" />

</xs:sequence>

</xs:complexType>

</xs:element>

You should now have a compiling schema which can be successfully tested against a serialised DataSet.

Transforms

If you're mapping a DataSet element between schemas, you'll need to use the Mass Copy Functoid to populate the target node from the contents of both the xs:any type elements on the source node:

This should give you a compiled map which you can test against a serialized instance. And if you have a .NET consumer on the other side of the mapped BizTalk output, it will correctly deserialize the response into a DataSet.

Processing Excel uploads with BizTalk and nServiceBus

EltonStoneman — Sat, 21 Nov 2009 01:57:56 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

Following on from the sample for processing Excel uploads with nServiceBus, I have a comparable version using BizTalk on MSDN Code Gallery here: BizTalk and nServiceBus Excel Upload. The BizTalk (2006 R2) sample uses a FILE receive port with a simple pipeline component to disassemble the Excel file into separate messages. A SQL send port subscribes to the messages, and with an outbound map calls the AdventureWorks stored procedure to insert each product.

Processing Excel in BizTalk is nothing new, but I wanted to do a comparison against the nServiceBus example, and also see how BizTalk and nServiceBus could be integrated. My original thinking was that the trigger in the nServiceBus solution relies on a FileSystemWatcher, which is less reliable and less flexible than BizTalk’s FILE adapter. A hybrid solution could use the FILE adapter to receive and parse the upload, then send each row as an AddProduct message to MSMQ, which the nServiceBus handler subscribes to.

Running this on the same environment (a Windows Server 2003 VM running under VirtualBox in Ubuntu 9.04), the BizTalk solution processes the 3,500 row Excel file in 3 mins 10 seconds (compared to 4m 15s for the distributed NSB running with 5 threads), and the 12,000 row file in 11m 14s (compared to 14m 0s). I was surprised to find the BizTalk solution running more quickly, as the original NSB was using non-recoverable messaging, so all messages were in memory, while BizTalk had the latency of writing to the message box.

Performance Comparison

This is not an intended to be a thorough benchmark of NSB and BizTalk – both sample projects are basic, untuned implementations, and the tests are on a single box rather than two or three. But for comparison, I ran the 3,500 row upload repeatedly under different configurations:

1. NSB with distributor – host and distributor using 20 threads

2. NSB with distributor – host and distributor using 20 threads, recoverable messaging

3. NSB without distributor – host using 20 threads

4. NSB without distributor – host using 20 threads, recoverable messaging

5. BizTalk FILE receive and SQL send

6. NSB parsing Excel file, BizTalk subscribing to AddProduct messages – MSMQ receive using batches of 100

7. BizTalk parsing Excel file, NSB without distributor subscribing to AddProduct messages – host using 20 threads

Which gave these averaged results:

Boosting the number of threads dramatically improved the NSB performance, and removing the distributor halved the duration. I would expect that running the distributor on a separate node would yield similarly good results on each processing node. The hybrid BizTalk/NSB configurations were the slowest – to be expected, as you have the latency of the message box and the latency of MSMQ saving to disk. NSB configurations without recoverable messaging were only marginally slower than the recoverable version, which seems incorrect based on Udi Dahan’s performance benchmark, so I’ll need to look into that further.

There’s plenty of scope for improving performance in both solutions. BizTalk can be endlessly tuned (the 2004 guidelines still apply as a good starting point). By modifying the NSB solution to send the AddProduct messages in one opration and using 80 threads for the handler, duration fell to 25 seconds – 140 messages per second.

Integrating BizTalk and nServiceBus

Despite having the worst performance, integrating BizTalk and nServiceBus is a viable option which may be very useful in some cases. It would allow you to leverage BizTalk’s adapter suite and mapping functionality to join LOB systems into an nServiceBus estate. Integration is actually very straightforward. By default NSB uses MSMQ, so to publish messages from BizTalk to an NSB subscriber just means configuring an MSMQ send port with the expected queue, and mapping your message to NSB format – which is an envelope containing one or messages:

<?xml version="1.0" ?>

<BatchSourcePath>E:\ExcelUpload\1.0.0.0\Binaries\Drops\ProductUpload.xls</BatchSourcePath>

<OriginatorDestination>ExcelUpload.Client.InputQueue@WIN2003R2-VM</OriginatorDestination>

<Name>new product 58</Name>

</AddProduct>

</Messages>

The MSMQ adapter lets you specify Recoverable and Transactional flags, so your BizTalk-generated messages have the same durability options. When an NSB handler is listening at the queue, it processes the BizTalk messages in the same way as NSB messages.

Publishing NSB messages to BizTalk is trickier, as the Send and Publish methods from NSB check to see if there are any subscribers before they write to the queue (compare this to the Notify method in RhinoServiceBus). If a BizTalk MSMQ receive port is the only handler, no subscribers will be registered with NSB and the message won’t be published. So you need to either call Send with a named queue (Bus.Send<AddProduct>("ExcelUpload.AddProductService.1.InputQueue", m =>...), or send a subscription message from BizTalk to register a subscriber. In the the second option you’d need to specify the message type in the subscription:

<?xml version="1.0" ?>

<string>ExcelUpload.Messages.StartBatchUpload, ExcelUpload.Messages, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null</string>

Running the Sample

(If you haven’t seen the original NSB sample, have a read as it details the pre-requisites).

To run the BizTalk solution, download ExcelUpload.BizTalk.Binaries.zip, unzip it and:

· Copy ExcelUpload.PipelineComponents.dll to your pipeline components directory – e.g. C:\Program Files\Microsoft BizTalk Server 2006\Pipeline Components

· If you haven’t done so from the previous sample, run uspInsertProduct.CREATE.sql

· Run uspInsertProduct.CREATESYNONYM.sql

· Import the MSI into BizTalk

· Run the MSI

· Start send port SendInsertProduct.SQL (you’ll need to change the connection details)

· Enable receive location ReceiveStartUpload.FILE.XLS (monitors c:\drops\ExcelUpload)

· Drop an Excel file

As before, you’ll need to clear down the database between runs if you drop the same file repeatedly.

If you want to run the hybrid solutions, you should use the binaries in the new download rather than the original, as these run without a distributor. Run start.cmd and there are three windows – Client (the file watcher), Host (the StartBatchUpload handler) and AddProductService (this is the AddProduct handler).

To run configuration 6 – NSB parsing the file and BizTalk handling the AddProduct messages:

· Run start.cmd

· Kill the AddProductService console

· Disable receive location ReceiveStartUpload.FILE.XLS

· Enable receive location ReceiveAddProduct.MSMQ

· Unenlist send port SendAddProduct.MSMQ

· Start send port SendInsertProduct.SQL

· Drop an Excel file

To run configuration 7 – BizTalk parsing the file and NSB handling the AddProduct messages:

· Run start.cmd

· Kill the Host and Client consoles

· Enable receive location ReceiveStartUpload.FILE.XLS

· Disable receive location ReceiveAddProduct.MSMQ

· Start send port SendAddProduct.MSMQ

· Unenlist send port SendInsertProduct.SQL

· Drop an Excel file

Source Code

The source and referenced assemblies are in ExcelUpload.BizTalk.Source.zip. There is a VS 2008 solution for the nServiceBus components, and a VS 2005 solution for the BizTalk components.

Using PowerShell in BizTalk Post-Processing Scripts

EltonStoneman — Mon, 08 Jun 2009 14:58:18 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

Often in BizTalk deployments you need to do additional work after installation. Typically your full install process may need to:

Install BizTalk artifact assemblies to the GAC
Install application dependencies to the GAC
Register an application source name in the registry, for logging to the Event Log
Create FILE send or receive locations on the local filesystem
Add application store configuration settings to Enterprise Single Sign-On (SSO)
Add log4net configuration settings to BTSNTSvc.exe.config

You can achieve this with a single BizTalk installer by configuring resources and post-processing scripts, and exporting an MSI from the application. Various scripting languages are supported in BizTalk installations (batch files, VBScript etc.), except the most logical – PowerShell, which gives first-class support for the filesystem, the registry, XML files and .NET objects. You can still use PowerShell by including scripts as resources, and using a batch file as the post-processing script, which acts as a harness to call the PowerShell scripts.

This walkthrough addresses all the points above. The completed BizTalk application is on MSDN Code Gallery here: BizTalk PowerShell Deployment Sample – import and install the MSI to deploy with the PowerShell script, or browse the ZIP file to see the scripts and resources.

1. Install BizTalk artifact assemblies to the GAC

This is straightforward, set the resource option "Add to the global assembly cache on MSI file install" to true – this happens by default if you add a BizTalk Assembly resource in the Administration Console:

Using the command line though, this is not the default option so you need to explicitly set –Options:GacOnInstall in BTSTask:

btstask AddResource

-ApplicationName:PowerShellSample

-Type:BizTalkAssembly

-Options:GacOnInstall

-Source:PowerShellSample.Schemas.dll

-Destination:%BTAD_InstallDir%\PowerShellSample.Schemas.dll

2. Install application dependencies to the GAC

As 1), except the resource type is System.BizTalk:Assembly (in BTSTask you can omit "System.BizTalk"). The command requires the same flag to add to the GAC on install:

btstask AddResource

-ApplicationName:PowerShellSample

-Type:Assembly

-Options:GacOnInstall

-Overwrite

-Source:.\Dependencies\SSOConfig.dll

-Destination:%BTAD_InstallDir%\Dependencies\SSOConfig.dll

In this case, I'm installing the SSOConfig assembly (from SSO Config Tool) which provides static .NET classes for accessing the SSO application configuration store. The Overwrite flag is set in case the resource already exists in another application.

3. Register an application source name in the registry, for logging to the Event Log

To log to the Application event log with your own source name, you need to add a registry key with the app name, and the name of the handler:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\Application\PowerShell.Sample

In PowerShell, this is done in using the New-Item cmdlet to create the key, and New-ItemProperty to set the key value:

New-Item -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Eventlog\Application\PowerShellSample' -Force

New-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Eventlog\Application\PowerShellSample' -Name 'EventMessageFile' -PropertyType ExpandString -Value 'C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\EventLogMessages.dll' –Force

(HKLM: is a PowerShell drive mapped to HKEY_LOCAL_MACHINE, and the –Force flag overwrites existing values).

To execute the PowerShell script on install, we need a batch file which BizTalk can run as a post-processing script. The batch file is very simple, just separating install and uninstall logic to individual PowerShell scripts, and redirecting script output to a log file:

cd "%BTAD_InstallDir%\Deployment"

if "%BTAD_InstallMode%" == "Install" ( powershell ".\PowerShellSample.Install.ps1" >> PowerShellSample.Install.ps1.log )

if "%BTAD_InstallMode%" == "Uninstall" ( powershell ".\ PowerShellSample.Uninstall.ps1" >> PowerShellSample.Uninstall.ps1.log )

Both the CMD and PS1 files need to be added as resources to the BizTalk application. The PS1 files are of type BizTalk:File, and the CMD harness is of type BizTalk:PostProcessingScript:

btstask AddResource

-ApplicationName:PowerShellSample

-Type:File

-Source:.\Deployment\PowerShellSample.Install.ps1

-Destination:%BTAD_InstallDir%\Deployment\PowerShellSample.Install.ps1

btstask AddResource

-ApplicationName:PowerShellSample

-Type:PostProcessingScript

-Source:.\Deployment\PowerShellSample.PostProcessing.cmd

-Destination:%BTAD_InstallDir%\Deployment\PowerShellSample.PostProcessing.cmd

4. Create FILE send or receive locations on the local filesystem

If you need to create static file locations, the same New-Item cmdlet is used with the filesystem provider. Specify the full path for the directory and any intermediate directories will be created if they don't exist. Use the –Force flag to suppress warnings if the directory already exists:

New-Item –Path 'c:\receiveLocations\x\y\z' -ItemType Directory –Force

Note that the resources in the BizTalk application are copies rather than references, so if you modify your PS1 files, you'll need to update the resource (in the Administration Console, select the resource and use Modify... Refresh, or re-run the BTSTask command).

5. Add settings to Enterprise Single Sign-On (SSO)

If you're using SSO to store group-wide application config, you can create or export an XML file of the settings using SSO Config Tool. We add the .ssoconfig file as a File resource to the application, then in the install script use PowerShell to call a .NET method to import the settings using the SSOConfig.SSOApplication class. The SSOConfig assembly is a resource which has already been deployed to the GAC by this point in the installation:

[Reflection.Assembly]::Load('SSOConfig, Version=1.1.0.0, Culture=neutral, PublicKeyToken=656a499478affdaf')

$configPath = [IO.Path]::Combine($env:BTAD_InstallDir, 'Deployment\PowerShellSample.ssoconfig')

$app = [SSOConfig.SSOApplication]::LoadFromXml($configPath)

$app.SaveToSSO()

Note that the PowerShell script has access to all the environment variables set by BizTalk on the install – accessed by prefixing $env: to the variable name, as we do here to get the installation directory from the installer ($env:BTAD_InstallDir).

6. Add settings to BTSNTSvc.exe.config

Modifying XML is straightforward in PowerShell. We want to configure an Event Log appender in the BizTalk service config file by adding the following XML:

</configSections>

</layout>

</appender>

<appender-ref ref="Sixeyed.CacheAdapter.EventLogAppender" />

</logger>

</log4net>

The Get-ItemProperty cmdlet can read the BizTalk install path from the registry, then Get-Content reads the file – casting it to XML for subsequent processing:

$installPath = Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\BizTalk Server\3.0' -Name 'InstallPath'

$btsConfigPath = [IO.Path]::Combine($installPath.InstallPath, 'BTSNTSvc.exe.config')

$xml = [xml] (Get-Content $btsConfigPath)

On a fresh install, the config file is quite bare and doesn't include a <configSections> element, so in that case we need to add both <configSections> and <log4net> nodes. We can't guarantee that other solutions haven't already modified the config file though, so <configSections> may exist, and <log4net> may also exist – in which case, we just need to add our specific appender and logger values (log4net allows you to define multiples of these in config, and we specify names which we can expect to be unique).

To achieve this, the script checks for each element first, creates it if it doesn't exist, then adds the specific settings:

$configSections = $xml.SelectSingleNode('configuration/configSections')

if ($configSections -eq $null)

{

$configSections = $xml.CreateElement('configSections')

$firstChild = $xml.configuration.get_FirstChild()

$xml.configuration.InsertBefore($configSections, $firstChild)

}

$log4netSection = $configSections.SelectSingleNode('section[@name="log4net"]')

if ($log4netSection -eq $null)

{

$log4netSection = $xml.CreateElement('section')

$log4netSection.SetAttribute('name', 'log4net')

$log4netSection.SetAttribute('type', 'log4net.Config.Log4NetConfigurationSectionHandler, log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821')

$configSections.AppendChild($log4netSection )

}

...

Finally the updates are saved over the original file:

$xml.Save($btsConfigPath)

Limitations

The main limitation with any post-processing script, is that the target environment selected for the install is not available. If you have multiple bindings files, the environment selected at runtime is only alive for the duration of the MSI import – the install has no reference to it, and there's no record made in the management database (not that I can see, please correct me if there is). This means you can't switch your script based on environment (e.g. to use different SSO config settings for System Test and Production). If that's a serious restriction you may prefer to create different MSIs per-environment in your build process, each containing the correct bindings file and scripts.

Specific to this approach, you need to have PowerShell installed on all the target machines, and configured to allow script execution (by default, scripts are not permitted to execute, as a security measure). Hopefully this is becoming the norm. Security also needs to be considered – the sample app writes to the registry and to SSO, so the installing context needs to have explicit permissions. The BizTalk installer runs under a separate security context from the installing user (by a trial-and-error process, this is NT AUTHORITY\ANONOYMOUS LOGON in my Server 2003 VM), so if you're amending SSO you'll need to set your SSO Administrators group membership correctly.

Benefits

The completed PowerShell scripts should be straightforward to read and maintain. All the post-installation requirements are implemented using a single technology, and many of the functions are reusable and could easily be parameterised and moved to a central script. The script is easy to test outside of the installer runtime, either manually using a batch file as a test harness (which sets up the relevant environment variables and then calls the post-processing file), or worked into an automated unit test.

The approach is not limited to BizTalk installations, so similar tasks for .NET deployments which are currently done with custom assemblies or Wix script can be isolated in the same way. With BizTalk and .NET installs using the same technology, you'll build up a library of high-quality, reusable PowerShell scripts.

I also like having the scripts deployed as part of the install, so in combination with the log files, you can see exactly what's been done to your environment and modify if necessary.

Extensions

With native cmdlets and community scripts, together with WMI, XML and .NET code, you can achieve any desired functionality with PowerShell scripts, and have them rapidly developed and tested. So you can easily add code to update version numbers in config files, remove your assemblies from the GAC on uninstall, access performance counters etc. And PowerShell scripts are just plain text so you can extract them into a T4 template and generate different scripts for different environments in your build process.

BizTalk Cache Adapter Samples

EltonStoneman — Mon, 01 Jun 2009 21:11:26 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

The sample BizTalk application provided with the BizTalk Cache Adapter on CodePlex illustrates three approaches for using the cache:

Simple Messaging – routes all messages configured for caching to the cache adapter, and sends cache-misses on to the service provider;
Orchestration – explicitly routes messages to either the cache adapter or service provider in orchestration logic;
Complex Messaging – routes messages configured for caching either to the cache adapter or to the service provider.

All the samples use FILE locations for the originating request and ultimate response. The expected locations will be created in the install, with sample request files in the receive locations. Enable the relevant receive location to run the sample, then copy the request file to the receive location again to re-run it – from the second run onwards, the responses should be the cached versions.

Installation

To use the samples, you'll need to install and configure a cache provider, the cache adapter, cache adapter sample, sample service, and the cache viewer (if you want to see the contents of the cache).

Both the Cache Adapter and Sample BizTalk MSIs use PowerShell scripts for their post-processing steps. If you want to install from the MSIs, you'll need to have PowerShell installed and set up to enable script execution (Set-ExecutionPolicy RemoteSigned is fine).

1. Cache Provider.

Download and install NCache Express (this is a free edition, you'll have a product key emailed to you), and configure a cache instance for the samples by modifying the configuration file (config.ncconf – by default in C:\Program Files\NCache Express\config):

<cache-config name="Sixeyed.CacheAdapter.NCacheExpressProvider.Tests" inproc="false">

<eviction-policy default-priority="normal" eviction-ratio="10%" eviction-enabled="true"/>

<perf-counters enabled="false"/>

</cache-config>

</configuration>

This creates a single-node cache on the local machine, limited to 5Mb of system memory. Start the cache by running the startcache tool (C:\Program Files\NCache Express\bin\tools):

startcache Sixeyed.CacheAdapter.NCacheExpressProvider.Tests

- and verify it's there with listcaches. You should see the cache listed as Running:

2. Cache Adapter.

Sixeyed.CacheAdapter.msi is the BizTalk MSI which installs the adapter and artifacts. Import the application into BizTalk and run the installer. Verify the Sixeyed.CacheAdapter application is listed, and add the adapter in the Administration Console through Platform Settings...Adapters...New.

Note: the installer adds the necessary adapter files to the GAC and sets up the registry keys. If you don't use the MSI, manual installation steps are: add the files from the Binaries release to a local directory and to the GAC; modify run the .REG command to specify assembly paths and import the .REG file; add the BizTalk artifact resources to your own application.

2. Cache Adapter Sample.

Sixeyed.CacheAdapterSample.msi is the BizTalk MSI which installs the sample application and artifacts, and configures the ports. Import the application into BizTalk and run the installer. Verify the Sixeyed.CacheAdapterSample application is listed, and that the message types are configured in SSO (check the contents of SSO application Sixeyed.CacheAdapter).

Note: the installer creates FILE send and receive locations, copies sample messages, and sets up SSO configuration for the cached message types. If you don't use the MSI, manual installation should follow the steps in: Source\Sixeyed.CacheAdapterSample\Deployment\Sixeyed.CacheAdapterSample.Install.ps1.

4. Sample Service.

Sixeyed.CacheAdapterSample.Service.msi installs a WCF service for the samples to call. Verify the service has deployed correctly by navigating to http://localhost/Sixeyed.CacheAdapterSample/SampleService.svc?wsdl.

5. Cache Viewer.

Sixeyed.CacheAdapter.CacheViewer.msi installs a Winforms app for viewing cache contents. Run the application and configure it to use the running cache instance, by navigating to the NCacheExpressProvider assembly and selecting the NCacheExpressProvider:

- and specifying the ID of the cache instance:

(This is the same management UI used for configuring cache adapter ports in the BizTalk Administration Console. For the samples the Cache Id is Sixeyed.CacheAdapter.NCacheExpressProvider.Tests).

Simple Messaging

Enable the receive location ReceiveRequest_SimpleMessaging.FILE and you should see the following workflow:

Message is picked up by the receive location, which is using the CacheXMLReceive pipeline. This checks the cache configuration in SSO, where the message type is configured for caching, so context property IsMessageCacheEnabled is set to true;
Send port SimpleMessaging_GetCache picks up the message and sends it to the cache adapter, using the request message to build a cache key, and checking for a response existing in the cache. There is no cached response, so the message returned from the cache adapter is the original request message, with context property IsCached set to false;
Send port SimpleMessaging_SendServiceRequest picks up the message and sends it to the service;
Send port SimpleMessaging_PutCache picks up the service response and adds it to the cache, using the cache key from the original request;
Send port SimpleMessaging_SendServiceResponse picks up the service response and copies it to the output directory.

Check the cache viewer and you will see a byte array cached with a GUID key:

The GUID is the hash of the request message, and the byte array is the response message. Subsequent drops of the same request into the Requests folder will find the cached response at step 2, and the response will be the cached message.

This workflow is suitable for any simple messaging solution, where static ports are used and where the original request and ultimate response can be auctioned by different ports. Configuration of the cache instance is contained in the send ports.

Orchestration

The orchestration sample follows the same workflow as the Simple Messaging sample, and uses the same CacheXMLReceive pipeline to add the caching properties to the incoming message. The orchestration sample makes the cache access with explicit send and receives, and configuration of the cache instance is still contained in the send ports.

Enable the receive location ReceiveRequest_Orchestration.FILE to run the sample. This is suitable for solutions where finer grained control is needed over the caching mechanism, and an orchestration is a viable option.

Complex Messaging

Both the simple implementations use the cache adapter as a passthrough when there is no matching response in the cache – the incoming message is returned by the cache adapter, and a send port picks it up and routes it on the service. This is not suitable for more complex solutions:

where the service is invoked through a dynamic port, as the outbound transport and location set on the original message are not reset by the cache adapter on the passed-through message;
where the originating receive is from a request-response port, as the interchange will accept the passed-through message as a response, and not return the actual service request.

For these scenarios, the cache checking can be moved forward into the originating receive, so the cache adapter is only called if the response is known to exist in the cache. Added complexity here is that the receive pipeline needs to have the cache instance configuration, along with the send ports. This is shown in the Complex Messaging sample – run by enabling the receive location ReceiveRequest_ComplexMessaging.FILE:

Message is picked up by the receive location, which is using a custom pipeline – CheckCacheReceive. This consists of the SetMessageCacheProperties component which checks SSO to see if the message type is configured for caching, followed by the CheckIsCached component which checks is the message response is actually in the cache, and sets the IsCached property:

Note that the cache instance is configured in the pipeline. Using a custom pipeline means you can configure the instance using the Cache Adapter UI in Visual Studio, which is easier and applies validation. While you could use a generic pipeline and modify the configuration per-instance on receive locations, that means manually hacking the XML;
1. If the message response is cached, send port ComplexMessaging_GetCache picks up the message and sends it to the cache adapter. The adapter is configured in the same way as other samples, and uses the same logic – but in this case the response is known to be cached, so it will be returned and the request message should never be passed-through;
2. If the message response is not cached, send port ComplexMessaging_SendServiceRequest calls the service provider passing the original request message;
3. Send port ComplexMessaging_PutCache picks up the service response and adds it to the cache, using the cache key from the original request;
4. Send port ComplexMessaging _SendServiceResponse picks up the service response and copies it to the output directory.

There is a risk in this scenario that the object will exist in cache at the point of checking, but been removed by the time the cache is actually read. In this case the response from step 2 will be a passthrough of the original request – in the sample, this means the originating request will be copied to the output directory. Suspending the message may be a better option, but depending on the solution you may be able to cope with this differently.

BizTalk Cache Adapter

EltonStoneman — Mon, 01 Jun 2009 21:08:31 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

In a recent project we had a requirement for a configurable cache, residing on a BizTalk host and storing responses from WCF services which the BizTalk app had brokered. Producing this as a generic cache adapter was the preferred option, but project timescales didn't allow for it – instead I've written the adapter as an open source component which is available on CodePlex: BizTalkCacheAdapter, and which we're now making use of in the project.

It's a simple design suitable for any situation where BizTalk is brokering services to consumers. The incoming request from the consumer is hashed to generate a cache key, and the response is stored in the cache against the request key. When future requests exactly match, they will receive the cached response; if the response doesn't exist in the cache, the service provider is invoked and the response is added to the cache. The adapter is independent of the cache store being used – currently the only option provided is NCache Express, but extending it to use memcached or Velocity will be straightforward.

Overview

The adapter uses context properties to store cache configuration, so it's suitable for orchestration or messaging solutions. Alongside the adapter on CodePlex is a sample project which demonstrates the approaches. The workflow is the same in both cases:

The consumer sends a service request via the CacheXMLReceive pipeline, which checks if the message type is configured for caching, and promotes the IsMessageCacheEnabled property;
If caching is enabled, a two-way message is sent to the cache adapter. This is a GET request, so the adapter checks the cache using the MessageCacheKey context property – if the context property is not available, the adapter writes it by hashing the contents of the service request in the message body;
The response from the cache adapter promotes the IsCached property, which identifies whether the response is in cache – if so, the body of the response message will be the cached service response, if not the body is the original service request which can be passed on to the service;
If IsCached is false, the service is invoked and a one-way message containing the response is sent to the cache adapter. This is a PUT request, so the adapter adds the message body to the cache using the MessageCacheKey;
The service response (either cached or from the provider) is returned to the consumer.

With the messaging solution, the same workflow is carried out using subscriptions to the promoted cache configuration properties, and the various send ports. This workflow may not be suitable for more complex messaging solutions, where services are invoked through dynamic send ports – there is an alternative workflow which I'll cover in a separate post.

The only real complexity in the adapter is in putting the service response into the cache – the outgoing service request message has the cache configuration properties promoted, but the incoming response does not. In orchestration solutions this isn't a problem, as the cache properties can be stored as state in the orchestration instance. In messaging solutions, the cache properties are retained by temporarily adding them to the cache, keyed by the interchange ID of the service instance, and removing them when the response is added to the cache.

Configuration

The cache adapter can be used with messages of any type, but with XML messages it can employ different cache configurations for different message types - e.g. a service to get current tax rates may configured to live for several days in the cache, while a service to get current exchange rates may use the same cache store, but be configured to expire after an hour. The CacheReceive and CacheReceiveXml pipelines contain properties for configuring the cache – it can be enabled/disabled for all messages, or (for the XML pipeline) only for configured message types:

Message-level configuration uses an SSO application store, which holds a MessageCacheConfigurationCollection object in XML:

<messageType>http://schemas.sixeyed.com/CacheAdapterSample/2009#GetServerDateTime</messageType>

</messageCacheConfiguration>

<messageType>http://schemas.xyz.com/2009#GetXYZ</messageType>

</messageCacheConfiguration>

...

Caching is enabled for a message if:

IsCacheEnabled is true on the pipeline AND
- IsMessageTypeConfigurationEnabled is false on the pipeline OR
- IsMessageTypeConfigurationEnabled is true on the pipeline AND
  - the incoming message type has an entry in the MessageCacheConfigurationCollection in SSO.

The lifespan of the object put into cache will be either the cacheLifespan specified in the MessageCacheConfiguration in SSO, or the defaultLifespan specified in the cache send port:

(The point at which the object is removed from cache will depend on the provider, some will remove it when its lifespan expires, others only when its lifespan expires and the cache is full or the cleanup schedule runs).

Installation

For version 1.0.0.0, the CodePlex project comes with three project releases, which can be used independently or together:

Sixeyed.CacheAdapter.msi - a BizTalk application MSI which can be imported and installed to your BizTalk server(s). The application contains the configuration schema and pipelines, so it can be used as-is, or the relevant artifacts can be moved to your own application;
Sixeyed.CacheAdapter.Binaries.zip – a ZIP file of the adapter binaries you can deploy manually. Use instead of the CacheAdapter MSI, if you don't want to deploy the whole BizTalk application;
Sixeyed.CacheAdapter.CacheViewer.msi – a Windows forms application which lets you view the contents of a cache store (the app uses the configuration components from the adapter management assembly, so it is also cache-provider agnostic):

- a documentation release:

Sixeyed.CacheAdapter.chm – documents the .NET assemblies of the adapter

- and two sample releases which should be deployed together:

Sixeyed.CacheAdapterSample.msi – a BizTalk application MSI which has sample usage of the cache adapter, demonstrating orchestration and messaging approaches;
Sixeyed.CacheAdapterSample.Service.msi – a WCF MSI which provides a service which the BizTalk app can cache.

Pre-requisite for the initial release is the non-commercial NCache Express to use as the cache provider.

To add the sample message-level cache configuration to SSO, the SSO Config Tool can import the provided .ssoconfig file into SSO.

I'll update the project to add a memcached provider, and write an updated post when it's available - and the same when Velocity is released. One advantage in having multiple providers would be in evaluating their performance against your expected caching load, so I'll look at extending the sample solution and adding some LoadGen scripts.

And as Michael Stephenson commented, a nice alternative would be to move the caching up to the WCF layer as a behavior, so WCF requests would check the cache first. Two options for this – it can either be a client behaviour using a local cache store (which could be more performant and would have less impact on server resources, but requires the cache provider to be deployed client-side), or as an operation behaviour using a shared server-side cache (which could be more performant, depending on the type of messages being cached, would impact on server and network resource, but would make the caching transparent to consumers). The complexity is on caching the response message if not already cached – the cache key from the original message will have been lost, so there needs to be a different approach for maintaining state to correlate the WCF service request and response.

Visualizing SOA’s Return on Investment

EltonStoneman — Thu, 26 Mar 2009 13:36:18 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

This is the rather grand title of a WebCast I've recorded recently which tries to illustrate where the return on investment comes after moving to a SOA strategy. The video is on Digital Forum: Visualizing SOA ROI , and the Open Source proof of concept it references is on CodePlex: ESBSimpleSamples.

In the WebCast I start with a blank Visual Studio solution and create a new Web app which consumes an existing service. All in it takes 3 lines of code and around 3 minutes of development (most of it spent correcting typos), and it doesn't require me to have any understanding of the underlying SOA infrastructure – in this case, Microsoft's ESB Guidance package sitting on BizTalk Server 2006 R2. Equally important, assuming the service we consume is in production, the testing and deployment effort needed for the new project is minimal.

This is a rework of a presentation I did with a client where I wanted to show how quick and easy it is to use their SOA implementation. I had reservations of showing a code demonstration to an audience which included IT heads (IT Director and Solution Architects) and project management, but it was surprisingly well received, so I've reproduced it using services and components which are publicly available.

Direct links to the references in the WebCast are here:

ESB Guidance: http://www.codeplex.com/esb
UK SOA/BPM User Group presentation: http://sbug.org.uk/media/p/129.aspx
Blog posts on ESB Guidance: http://geekswithblogs.net/EltonStoneman/category/7947.aspx

Using WCF-Custom Bindings in Dynamic Ports

EltonStoneman — Fri, 06 Mar 2009 17:38:49 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

BizTalk 2006 R2 ships with WCF adapters and pre-configured settings for common bindings - basicHttp and wsHttp being typically used for SOAP messaging. With a static port you can use the WCF-Custom adapter, select an existing binding and configure it further in the UI, with the full set of binding options available to you:

Here I'm using basicHttp, but I've configured the maxReceivedMessageSize, sendTimeout and transferMode settings to allow us to call long-running WCF services which return large responses.

In a dynamic port, you're limited to the settings you can configure, as the WCF Adapter Property Schema doesn't contain the full set of binding properties. You can set the sendTimeout and maxReceivedMessageSize on an outgoing message using code:

port(Microsoft.XLANGs.BaseTypes.TransportType) = "WCF-BasicHttp";

requestMessage(WCF.MaxReceivedMessageSize) = 104857600;

requestMessage(WCF.SendTimeout) = "00:10:00";

- but there's no way to access less common properties like transferMode if you're using this approach. Originally this is how we were configuring our outgoing messages, using SSO to store the values used for the binding properties (see Receiving large WCF response messages in ESB Guidance), but we soon ran into an issue with one of the BizTalk servers running out of memory when attempting to process a large WCF response message:

System.InsufficientMemoryException: Failed to allocate a managed memory buffer of 104857600 bytes. The amount of available memory may be low. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

at System.ServiceModel.Diagnostics.Utility.AllocateByteArray(Int32 size)

For basicHttp, the transferMode setting allows you to specify that messages should be streamed (one way or both ways), or buffered - the default is buffered, so when we receive the response message, although it will be streamed in the BizTalk stack, it is completely loaded into memory by the WCF stack. Note that WCF is trying to allocate the full value specified in maxReceivedMessageSize – 100 Mb – even though we've set this as a practical maximum, and the actual incoming message was half that size.

To remedy it, I've switched to using the WCF-Custom transport, and specifying basicHttp with my additional settings in the BindingConfiguration property:

port(Microsoft.XLANGs.BaseTypes.TransportType) = "WCF-Custom";

requestMessage(WCF.BindingType) = "basicHttpBinding";

requestMessage(WCF.BindingConfiguration) = "<binding name=\"basicHttpBinding\" sendTimeout=\"00:10:00\" maxReceivedMessageSize=\"104857600\" />";

The BindingType needs to be specified in addition to the configuration. For the BindingConfiguration value, the Properties page for a static WCF-Custom port allows you to export the settings, so you can configure it in the UI and then save the XML representation, rather than coding it all by hand.

By shifting the BindingConfiguration value to the SSO config store, we'll have the full range of WCF configuration available to change at run-time, so a switch to using wsHttp or netTcp bindings is just a setting change using the SSO Config Tool, and a change to the endpoint address (which we configure in UDDI in this case). Originally I wanted finer control over what could be configured, so we could limit changes to a known set of properties. But having tried this approach I prefer it – it means switching bindings and adding or changing any settings can easily be done without modifying the solution – and I'll be recommending it as best practice.

MSDTC Issue with Clustered BizTalk Host

EltonStoneman — Mon, 23 Feb 2009 17:48:55 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

We had an issue with one of our BizTalk estates with incoming messages being suspended if they were bigger than the large message threshold:

A response message sent to adapter "WCF-BasicHttp" on Receive Location: "x" with URI:"y" is suspended. Error details: There was a failure executing the response(receive) pipeline: "z" Source: "Unknown " Send Port: "x" URI: "y" Reason: 0x8004d027

The error message wasn't particularly helpful but the reason code 0x8004d027 had a few threads and posts relating to DTC. This made sense as we're in a distributed environment so DTC will be used to create message fragments under a transaction, if the whole message exceeds the threshold size. Using Web Service Studio I found the expected message size, and setting the large message threshold above that figure fixed the problem so that seemed to confirm it.

We finally tracked the issue down to permissions on the cluster where MSDTC was running. Buried in the Event Log for the BizTalk host suspending the messages was this error:

The application could not connect to MSDTC because of insufficient permissions. Please make sure that the identity under which the application is running has permission to access the cluster. Please refer to MSCS documentation on how to grant permissions. Error Specifics: d:\nt\com\complus\dtc\dtc\msdtcprx\src\dtcinit.cpp:652, Pid: 6772 No Callstack, CmdLine: "C:\Program Files\Microsoft BizTalk Server 2006\BTSNTSvc.exe" -group "BizTalk Group" -name "x" -btsapp "y"

This was logged when the BizTalk service started, which had been the day before we found the suspended messages, so I didn't see the connection. This post from Ben Cops: http://bencops.blogspot.com/2008/12/error-on-installing-sso-on-clustered.html pointed me in the right direction, and the steps for adding permissions are simple - the only oddity is that you set permissions at the cluster level, not at the resource level:

Open Cluster Administrator
Right-click the cluster (the topmost node) and select Properties
Under Security, add the BizTalk user group with Full Control

This and a host restart fixed the issue. Along the way we did notice a couple of other issues in our MSDTC configuration which we tracked down and fixed with DTCPing. Worth noting that in Windows Server 2003 the properties for MSDTC under dcomcnfg may not correctly reflect the registry settings, so while you think DTC security is configured like this:

- it may actually be configured like this:

The DTCPing log will tell you what registry settings are in use, and the BizTalk Best Practices Analyzer will warn you if the DTC security settings are incorrect, so worth double-checking this.

An Infrastructure Design for ESB Guidance

EltonStoneman — Mon, 02 Feb 2009 08:08:46 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

Our ESB Guidance project is nearing code-completion and we're running System Test on an infrastructure which is suitable for production use. Designed for performance, reliability and scalability, we use servers in a combination of NLB farms, Windows clusters and SQL clusters. Shown below is a modified version of the layout, focusing on the BizTalk landscape:

A few of the design decisions are worth looking at:

Coordination Services run a host for Scheduled Task and FTP adapters, building itinerary requests from received messages. Both adapters are unable to flag an incoming message as being in-process, so are not safe on multiple instances – we have two servers in an active/passive Windows cluster to provide redundancy;
Itinerary Services run a host for incoming itinerary requests from the SOAP adapter. This is instance-safe, so we have two servers configured in a farm for redundancy and performance;
Itinerary Processing runs a host for the custom orchestration we have for handling calls to WCF service providers (not shown in this diagram, but running on a separate Web farm and SQL instance). All BizTalk servers run instances of this host, so any with capacity can service calls;
Enterprise Single Sign-On is located it on the Windows cluster with the Coordination Services, as it cannot run under an NLB farm. The Coordination Service is expected to have spare capacity, so hosting SSO shouldn't impact performance;
UDDI Services run on the BizTalk NLB farm with the Itinerary Services. A small compromise here from the ideal of having a separate UDDI host, but resolver calls will all be made during Itinerary Processing and we use a custom UDDI resolver which holds the cache indefinitely (until manually invalidated when a change is promoted to the UDDI Service), so after the initial warm-up, UDDI Services will have negligible usage;
UDDI Database runs on the same SQL instance as the BizTalk databases. The second small compromise – breaking the cardinal rule that BizTalk should have a dedicated SQL instance with dedicated hardware… As UDDI calls will be minimal, this is a worthwhile performance trade to get the redundancy of having UDDI on the SQL cluster.

The design is intended to cope with a relatively low initial throughput – 250,000 messages in the first year, 98% of which will be a few kilobytes, 2% running into megabytes. BizTalk servers all running Enterprise Edition in a Group, with just Microsoft.Practices.ESB and our custom application installed. Currently the BizTalk servers are running single quad-core CPUs with 4Gb RAM.

Our next task is to run some simulated load through the setup and modify the host layout or physical capacity if necessary. I'll cover the load testing results in a future post.

BizTalk Deployments with PowerShell

EltonStoneman — Wed, 10 Dec 2008 23:09:56 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

As part of my first look at ESB Guidance 2.0 (which has some excellent enhancements which I'll post about shortly) I've been interested in the way the sample solutions are deployed. The P&P team have put together PowerShell scripts which take care of the usual things you'd find in a deployment scenario – stopping services, creating applications, adding resources and bindings.

In a Continuous Integration environment, you'd typically want to deploy to a BizTalk instance as part of the build process, to ensure the application configuration is valid, and as a prerequisite for running unit and integration tests. Currently I do this with MSBuild scripts using a variety of community, Sdc and custom tasks, and command-line tools. It makes for fragmented scripts that are difficult to read and maintain, and can be fragile to execute.

In the PowerShell version a central script isolates the complexity for common functions, and then the individual install scripts use a neat, readable and hugely intuitive syntax. This is a complete install script for one of the sample solutions (MultipleWebServices_install.ps1), which builds and then deploys the artifacts:

$relativePath = "..\.."

$samplesPath = $solutionPath+"\Samples\MultipleWebServices"

. ..\..\..\..\Core\Install\Scripts\ESBFunctions.ps1

function BuildItinerarySolution

{

param([System.String] $buildType)

&$FrameworkPath"\v3.5\MSBuild.exe" $relativePath"\GlobalBank.ESB.Samples.MultipleWebServices.sln" /t:Rebuild /p:Configuration=$buildType

}

BuildItinerarySolution Debug

StopBTSApplication "Microsoft.Practices.ESB"

StopBTSApplication "GlobalBank.ESB"

StopBTSServices

ImportBTSResource "GlobalBank.ESB" "BizTalkAssembly" $relativePath"\Source\ESB.MultipleWebServices.Maps\bin\Debug\ESB.MultipleWebServices.Maps.dll"

ImportBTSResource "GlobalBank.ESB" "BizTalkAssembly" $relativePath"\Source\ESB.MultipleWebServices.Orchestrations\bin\Debug\ESB.MultipleWebServices.Orchestrations.dll"

GacOperation "-if" $relativePath"\Source\ESB.MultipleWebServices.Maps\bin\Debug\ESB.MultipleWebServices.Maps.dll"

GacOperation "-if" $relativePath"\Source\ESB.MultipleWebServices.Orchestrations\bin\Debug\ESB.MultipleWebServices.Orchestrations.dll"

ImportBinding "GlobalBank.ESB" $relativePath\Install\Binding\GlobalBank.ESB.MultipleWebServices_Bindings.xml

StartBTSApplication "Microsoft.Practices.ESB"

StartBTSApplication "GlobalBank.ESB"

StartBTSServices

The common script (ESBFunctions.ps1) defines the main functions – ImportBTSResource etc. – so the logic in the individual scripts is very clean. Common functions make use of PowerShell commands, the BizTalk Explorer OM, BTSTask and custom tools as appropriate, so there's still a variety of tools under the covers, but the common script is a facade over them all. This makes the install scripts neater and abstracts them from the actual implementation of the functions. Below are snippets for the functions to stop and to remove a BizTalk application:

function StopBTSApplication

{

param([string]$appName)

$exp = New-Object Microsoft.BizTalk.ExplorerOM.BtsCatalogExplorer

$exp.ConnectionString = $btsConnectionString

$app = $exp.Applications[$appName]

if($app -eq $null)

{

Write-Host "Application " $appName " not found" -fore Red

}

else

{

if($app.Status -ne 2)

{

#full stop of application

$null = $app.Stop(63)

$null = $exp.SaveChanges()

Write-Host "Stopped application: " $appName

}

function RemoveBTSApplication

{

param([string]$appName)

Write-Output "Removing BTS Application "+ $appName

BTSTask.exe RemoveApp /A:$appName

}

There's some complexity in these common functions, but they're all tried and tested by the Patterns & Practices team. This is a new approach for the ESB Guidance installs (v1.0 used a combination of batch files to deploy the samples), but I don't think there are any inherent dependencies on BTS 2009 so this approach and the common script should work for BTS 2006 solutions. I'm going to look at migrating my current build scripts to this approach as it's so much simpler to work with, and the execution doesn't rely on chaining together unrelated tasks in MSBuild.

The ESBG samples specifically name resources to deploy, whereas my current scripts run dynamically over whichever artifacts they find, but Bart De Smet has a nice MSBuild task for running PowerShell scripts, which takes parameters so that's one option, and generating PS scripts using the ExecuteT4Template task is another.

If I do get this running for BizTalk 2006 R2 solutions, I'll post a working set of sample scripts.