BizTalk 2006 R2

Using PowerShell in BizTalk Post-Processing Scripts

EltonStoneman — Mon, 08 Jun 2009 20:58:18 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

Often in BizTalk deployments you need to do additional work after installation. Typically your full install process may need to:

Install BizTalk artifact assemblies to the GAC
Install application dependencies to the GAC
Register an application source name in the registry, for logging to the Event Log
Create FILE send or receive locations on the local filesystem
Add application store configuration settings to Enterprise Single Sign-On (SSO)
Add log4net configuration settings to BTSNTSvc.exe.config

You can achieve this with a single BizTalk installer by configuring resources and post-processing scripts, and exporting an MSI from the application. Various scripting languages are supported in BizTalk installations (batch files, VBScript etc.), except the most logical – PowerShell, which gives first-class support for the filesystem, the registry, XML files and .NET objects. You can still use PowerShell by including scripts as resources, and using a batch file as the post-processing script, which acts as a harness to call the PowerShell scripts.

This walkthrough addresses all the points above. The completed BizTalk application is on MSDN Code Gallery here: BizTalk PowerShell Deployment Sample – import and install the MSI to deploy with the PowerShell script, or browse the ZIP file to see the scripts and resources.

1. Install BizTalk artifact assemblies to the GAC

This is straightforward, set the resource option "Add to the global assembly cache on MSI file install" to true – this happens by default if you add a BizTalk Assembly resource in the Administration Console:

Using the command line though, this is not the default option so you need to explicitly set –Options:GacOnInstall in BTSTask:

btstask AddResource

-ApplicationName:PowerShellSample

-Type:BizTalkAssembly

-Options:GacOnInstall

-Source:PowerShellSample.Schemas.dll

-Destination:%BTAD_InstallDir%\PowerShellSample.Schemas.dll

2. Install application dependencies to the GAC

As 1), except the resource type is System.BizTalk:Assembly (in BTSTask you can omit "System.BizTalk"). The command requires the same flag to add to the GAC on install:

btstask AddResource

-ApplicationName:PowerShellSample

-Type:Assembly

-Options:GacOnInstall

-Overwrite

-Source:.\Dependencies\SSOConfig.dll

-Destination:%BTAD_InstallDir%\Dependencies\SSOConfig.dll

In this case, I'm installing the SSOConfig assembly (from SSO Config Tool) which provides static .NET classes for accessing the SSO application configuration store. The Overwrite flag is set in case the resource already exists in another application.

3. Register an application source name in the registry, for logging to the Event Log

To log to the Application event log with your own source name, you need to add a registry key with the app name, and the name of the handler:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\Application\PowerShell.Sample

In PowerShell, this is done in using the New-Item cmdlet to create the key, and New-ItemProperty to set the key value:

New-Item -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Eventlog\Application\PowerShellSample' -Force

New-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Eventlog\Application\PowerShellSample' -Name 'EventMessageFile' -PropertyType ExpandString -Value 'C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\EventLogMessages.dll' –Force

(HKLM: is a PowerShell drive mapped to HKEY_LOCAL_MACHINE, and the –Force flag overwrites existing values).

To execute the PowerShell script on install, we need a batch file which BizTalk can run as a post-processing script. The batch file is very simple, just separating install and uninstall logic to individual PowerShell scripts, and redirecting script output to a log file:

cd "%BTAD_InstallDir%\Deployment"

if "%BTAD_InstallMode%" == "Install" ( powershell ".\PowerShellSample.Install.ps1" >> PowerShellSample.Install.ps1.log )

if "%BTAD_InstallMode%" == "Uninstall" ( powershell ".\ PowerShellSample.Uninstall.ps1" >> PowerShellSample.Uninstall.ps1.log )

Both the CMD and PS1 files need to be added as resources to the BizTalk application. The PS1 files are of type BizTalk:File, and the CMD harness is of type BizTalk:PostProcessingScript:

btstask AddResource

-ApplicationName:PowerShellSample

-Type:File

-Source:.\Deployment\PowerShellSample.Install.ps1

-Destination:%BTAD_InstallDir%\Deployment\PowerShellSample.Install.ps1

btstask AddResource

-ApplicationName:PowerShellSample

-Type:PostProcessingScript

-Source:.\Deployment\PowerShellSample.PostProcessing.cmd

-Destination:%BTAD_InstallDir%\Deployment\PowerShellSample.PostProcessing.cmd

4. Create FILE send or receive locations on the local filesystem

If you need to create static file locations, the same New-Item cmdlet is used with the filesystem provider. Specify the full path for the directory and any intermediate directories will be created if they don't exist. Use the –Force flag to suppress warnings if the directory already exists:

New-Item –Path 'c:\receiveLocations\x\y\z' -ItemType Directory –Force

Note that the resources in the BizTalk application are copies rather than references, so if you modify your PS1 files, you'll need to update the resource (in the Administration Console, select the resource and use Modify... Refresh, or re-run the BTSTask command).

5. Add settings to Enterprise Single Sign-On (SSO)

If you're using SSO to store group-wide application config, you can create or export an XML file of the settings using SSO Config Tool. We add the .ssoconfig file as a File resource to the application, then in the install script use PowerShell to call a .NET method to import the settings using the SSOConfig.SSOApplication class. The SSOConfig assembly is a resource which has already been deployed to the GAC by this point in the installation:

[Reflection.Assembly]::Load('SSOConfig, Version=1.1.0.0, Culture=neutral, PublicKeyToken=656a499478affdaf')

$configPath = [IO.Path]::Combine($env:BTAD_InstallDir, 'Deployment\PowerShellSample.ssoconfig')

$app = [SSOConfig.SSOApplication]::LoadFromXml($configPath)

$app.SaveToSSO()

Note that the PowerShell script has access to all the environment variables set by BizTalk on the install – accessed by prefixing $env: to the variable name, as we do here to get the installation directory from the installer ($env:BTAD_InstallDir).

6. Add settings to BTSNTSvc.exe.config

Modifying XML is straightforward in PowerShell. We want to configure an Event Log appender in the BizTalk service config file by adding the following XML:

</configSections>

</layout>

</appender>

<appender-ref ref="Sixeyed.CacheAdapter.EventLogAppender" />

</logger>

</log4net>

The Get-ItemProperty cmdlet can read the BizTalk install path from the registry, then Get-Content reads the file – casting it to XML for subsequent processing:

$installPath = Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\BizTalk Server\3.0' -Name 'InstallPath'

$btsConfigPath = [IO.Path]::Combine($installPath.InstallPath, 'BTSNTSvc.exe.config')

$xml = [xml] (Get-Content $btsConfigPath)

On a fresh install, the config file is quite bare and doesn't include a <configSections> element, so in that case we need to add both <configSections> and <log4net> nodes. We can't guarantee that other solutions haven't already modified the config file though, so <configSections> may exist, and <log4net> may also exist – in which case, we just need to add our specific appender and logger values (log4net allows you to define multiples of these in config, and we specify names which we can expect to be unique).

To achieve this, the script checks for each element first, creates it if it doesn't exist, then adds the specific settings:

$configSections = $xml.SelectSingleNode('configuration/configSections')

if ($configSections -eq $null)

{

$configSections = $xml.CreateElement('configSections')

$firstChild = $xml.configuration.get_FirstChild()

$xml.configuration.InsertBefore($configSections, $firstChild)

}

$log4netSection = $configSections.SelectSingleNode('section[@name="log4net"]')

if ($log4netSection -eq $null)

{

$log4netSection = $xml.CreateElement('section')

$log4netSection.SetAttribute('name', 'log4net')

$log4netSection.SetAttribute('type', 'log4net.Config.Log4NetConfigurationSectionHandler, log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821')

$configSections.AppendChild($log4netSection )

}

...

Finally the updates are saved over the original file:

$xml.Save($btsConfigPath)

Limitations

The main limitation with any post-processing script, is that the target environment selected for the install is not available. If you have multiple bindings files, the environment selected at runtime is only alive for the duration of the MSI import – the install has no reference to it, and there's no record made in the management database (not that I can see, please correct me if there is). This means you can't switch your script based on environment (e.g. to use different SSO config settings for System Test and Production). If that's a serious restriction you may prefer to create different MSIs per-environment in your build process, each containing the correct bindings file and scripts.

Specific to this approach, you need to have PowerShell installed on all the target machines, and configured to allow script execution (by default, scripts are not permitted to execute, as a security measure). Hopefully this is becoming the norm. Security also needs to be considered – the sample app writes to the registry and to SSO, so the installing context needs to have explicit permissions. The BizTalk installer runs under a separate security context from the installing user (by a trial-and-error process, this is NT AUTHORITY\ANONOYMOUS LOGON in my Server 2003 VM), so if you're amending SSO you'll need to set your SSO Administrators group membership correctly.

Benefits

The completed PowerShell scripts should be straightforward to read and maintain. All the post-installation requirements are implemented using a single technology, and many of the functions are reusable and could easily be parameterised and moved to a central script. The script is easy to test outside of the installer runtime, either manually using a batch file as a test harness (which sets up the relevant environment variables and then calls the post-processing file), or worked into an automated unit test.

The approach is not limited to BizTalk installations, so similar tasks for .NET deployments which are currently done with custom assemblies or Wix script can be isolated in the same way. With BizTalk and .NET installs using the same technology, you'll build up a library of high-quality, reusable PowerShell scripts.

I also like having the scripts deployed as part of the install, so in combination with the log files, you can see exactly what's been done to your environment and modify if necessary.

Extensions

With native cmdlets and community scripts, together with WMI, XML and .NET code, you can achieve any desired functionality with PowerShell scripts, and have them rapidly developed and tested. So you can easily add code to update version numbers in config files, remove your assemblies from the GAC on uninstall, access performance counters etc. And PowerShell scripts are just plain text so you can extract them into a T4 template and generate different scripts for different environments in your build process.

BizTalk Cache Adapter Samples

EltonStoneman — Tue, 02 Jun 2009 03:11:26 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

The sample BizTalk application provided with the BizTalk Cache Adapter on CodePlex illustrates three approaches for using the cache:

Simple Messaging – routes all messages configured for caching to the cache adapter, and sends cache-misses on to the service provider;
Orchestration – explicitly routes messages to either the cache adapter or service provider in orchestration logic;
Complex Messaging – routes messages configured for caching either to the cache adapter or to the service provider.

All the samples use FILE locations for the originating request and ultimate response. The expected locations will be created in the install, with sample request files in the receive locations. Enable the relevant receive location to run the sample, then copy the request file to the receive location again to re-run it – from the second run onwards, the responses should be the cached versions.

Installation

To use the samples, you'll need to install and configure a cache provider, the cache adapter, cache adapter sample, sample service, and the cache viewer (if you want to see the contents of the cache).

Both the Cache Adapter and Sample BizTalk MSIs use PowerShell scripts for their post-processing steps. If you want to install from the MSIs, you'll need to have PowerShell installed and set up to enable script execution (Set-ExecutionPolicy RemoteSigned is fine).

1. Cache Provider.

Download and install NCache Express (this is a free edition, you'll have a product key emailed to you), and configure a cache instance for the samples by modifying the configuration file (config.ncconf – by default in C:\Program Files\NCache Express\config):

<cache-config name="Sixeyed.CacheAdapter.NCacheExpressProvider.Tests" inproc="false">

<eviction-policy default-priority="normal" eviction-ratio="10%" eviction-enabled="true"/>

<perf-counters enabled="false"/>

</cache-config>

</configuration>

This creates a single-node cache on the local machine, limited to 5Mb of system memory. Start the cache by running the startcache tool (C:\Program Files\NCache Express\bin\tools):

startcache Sixeyed.CacheAdapter.NCacheExpressProvider.Tests

- and verify it's there with listcaches. You should see the cache listed as Running:

2. Cache Adapter.

Sixeyed.CacheAdapter.msi is the BizTalk MSI which installs the adapter and artifacts. Import the application into BizTalk and run the installer. Verify the Sixeyed.CacheAdapter application is listed, and add the adapter in the Administration Console through Platform Settings...Adapters...New.

Note: the installer adds the necessary adapter files to the GAC and sets up the registry keys. If you don't use the MSI, manual installation steps are: add the files from the Binaries release to a local directory and to the GAC; modify run the .REG command to specify assembly paths and import the .REG file; add the BizTalk artifact resources to your own application.

2. Cache Adapter Sample.

Sixeyed.CacheAdapterSample.msi is the BizTalk MSI which installs the sample application and artifacts, and configures the ports. Import the application into BizTalk and run the installer. Verify the Sixeyed.CacheAdapterSample application is listed, and that the message types are configured in SSO (check the contents of SSO application Sixeyed.CacheAdapter).

Note: the installer creates FILE send and receive locations, copies sample messages, and sets up SSO configuration for the cached message types. If you don't use the MSI, manual installation should follow the steps in: Source\Sixeyed.CacheAdapterSample\Deployment\Sixeyed.CacheAdapterSample.Install.ps1.

4. Sample Service.

Sixeyed.CacheAdapterSample.Service.msi installs a WCF service for the samples to call. Verify the service has deployed correctly by navigating to http://localhost/Sixeyed.CacheAdapterSample/SampleService.svc?wsdl.

5. Cache Viewer.

Sixeyed.CacheAdapter.CacheViewer.msi installs a Winforms app for viewing cache contents. Run the application and configure it to use the running cache instance, by navigating to the NCacheExpressProvider assembly and selecting the NCacheExpressProvider:

- and specifying the ID of the cache instance:

(This is the same management UI used for configuring cache adapter ports in the BizTalk Administration Console. For the samples the Cache Id is Sixeyed.CacheAdapter.NCacheExpressProvider.Tests).

Simple Messaging

Enable the receive location ReceiveRequest_SimpleMessaging.FILE and you should see the following workflow:

Message is picked up by the receive location, which is using the CacheXMLReceive pipeline. This checks the cache configuration in SSO, where the message type is configured for caching, so context property IsMessageCacheEnabled is set to true;
Send port SimpleMessaging_GetCache picks up the message and sends it to the cache adapter, using the request message to build a cache key, and checking for a response existing in the cache. There is no cached response, so the message returned from the cache adapter is the original request message, with context property IsCached set to false;
Send port SimpleMessaging_SendServiceRequest picks up the message and sends it to the service;
Send port SimpleMessaging_PutCache picks up the service response and adds it to the cache, using the cache key from the original request;
Send port SimpleMessaging_SendServiceResponse picks up the service response and copies it to the output directory.

Check the cache viewer and you will see a byte array cached with a GUID key:

The GUID is the hash of the request message, and the byte array is the response message. Subsequent drops of the same request into the Requests folder will find the cached response at step 2, and the response will be the cached message.

This workflow is suitable for any simple messaging solution, where static ports are used and where the original request and ultimate response can be auctioned by different ports. Configuration of the cache instance is contained in the send ports.

Orchestration

The orchestration sample follows the same workflow as the Simple Messaging sample, and uses the same CacheXMLReceive pipeline to add the caching properties to the incoming message. The orchestration sample makes the cache access with explicit send and receives, and configuration of the cache instance is still contained in the send ports.

Enable the receive location ReceiveRequest_Orchestration.FILE to run the sample. This is suitable for solutions where finer grained control is needed over the caching mechanism, and an orchestration is a viable option.

Complex Messaging

Both the simple implementations use the cache adapter as a passthrough when there is no matching response in the cache – the incoming message is returned by the cache adapter, and a send port picks it up and routes it on the service. This is not suitable for more complex solutions:

where the service is invoked through a dynamic port, as the outbound transport and location set on the original message are not reset by the cache adapter on the passed-through message;
where the originating receive is from a request-response port, as the interchange will accept the passed-through message as a response, and not return the actual service request.

For these scenarios, the cache checking can be moved forward into the originating receive, so the cache adapter is only called if the response is known to exist in the cache. Added complexity here is that the receive pipeline needs to have the cache instance configuration, along with the send ports. This is shown in the Complex Messaging sample – run by enabling the receive location ReceiveRequest_ComplexMessaging.FILE:

Message is picked up by the receive location, which is using a custom pipeline – CheckCacheReceive. This consists of the SetMessageCacheProperties component which checks SSO to see if the message type is configured for caching, followed by the CheckIsCached component which checks is the message response is actually in the cache, and sets the IsCached property:

Note that the cache instance is configured in the pipeline. Using a custom pipeline means you can configure the instance using the Cache Adapter UI in Visual Studio, which is easier and applies validation. While you could use a generic pipeline and modify the configuration per-instance on receive locations, that means manually hacking the XML;
1. If the message response is cached, send port ComplexMessaging_GetCache picks up the message and sends it to the cache adapter. The adapter is configured in the same way as other samples, and uses the same logic – but in this case the response is known to be cached, so it will be returned and the request message should never be passed-through;
2. If the message response is not cached, send port ComplexMessaging_SendServiceRequest calls the service provider passing the original request message;
3. Send port ComplexMessaging_PutCache picks up the service response and adds it to the cache, using the cache key from the original request;
4. Send port ComplexMessaging _SendServiceResponse picks up the service response and copies it to the output directory.

There is a risk in this scenario that the object will exist in cache at the point of checking, but been removed by the time the cache is actually read. In this case the response from step 2 will be a passthrough of the original request – in the sample, this means the originating request will be copied to the output directory. Suspending the message may be a better option, but depending on the solution you may be able to cope with this differently.

BizTalk Cache Adapter

EltonStoneman — Tue, 02 Jun 2009 03:08:31 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

In a recent project we had a requirement for a configurable cache, residing on a BizTalk host and storing responses from WCF services which the BizTalk app had brokered. Producing this as a generic cache adapter was the preferred option, but project timescales didn't allow for it – instead I've written the adapter as an open source component which is available on CodePlex: BizTalkCacheAdapter, and which we're now making use of in the project.

It's a simple design suitable for any situation where BizTalk is brokering services to consumers. The incoming request from the consumer is hashed to generate a cache key, and the response is stored in the cache against the request key. When future requests exactly match, they will receive the cached response; if the response doesn't exist in the cache, the service provider is invoked and the response is added to the cache. The adapter is independent of the cache store being used – currently the only option provided is NCache Express, but extending it to use memcached or Velocity will be straightforward.

Overview

The adapter uses context properties to store cache configuration, so it's suitable for orchestration or messaging solutions. Alongside the adapter on CodePlex is a sample project which demonstrates the approaches. The workflow is the same in both cases:

The consumer sends a service request via the CacheXMLReceive pipeline, which checks if the message type is configured for caching, and promotes the IsMessageCacheEnabled property;
If caching is enabled, a two-way message is sent to the cache adapter. This is a GET request, so the adapter checks the cache using the MessageCacheKey context property – if the context property is not available, the adapter writes it by hashing the contents of the service request in the message body;
The response from the cache adapter promotes the IsCached property, which identifies whether the response is in cache – if so, the body of the response message will be the cached service response, if not the body is the original service request which can be passed on to the service;
If IsCached is false, the service is invoked and a one-way message containing the response is sent to the cache adapter. This is a PUT request, so the adapter adds the message body to the cache using the MessageCacheKey;
The service response (either cached or from the provider) is returned to the consumer.

With the messaging solution, the same workflow is carried out using subscriptions to the promoted cache configuration properties, and the various send ports. This workflow may not be suitable for more complex messaging solutions, where services are invoked through dynamic send ports – there is an alternative workflow which I'll cover in a separate post.

The only real complexity in the adapter is in putting the service response into the cache – the outgoing service request message has the cache configuration properties promoted, but the incoming response does not. In orchestration solutions this isn't a problem, as the cache properties can be stored as state in the orchestration instance. In messaging solutions, the cache properties are retained by temporarily adding them to the cache, keyed by the interchange ID of the service instance, and removing them when the response is added to the cache.

Configuration

The cache adapter can be used with messages of any type, but with XML messages it can employ different cache configurations for different message types - e.g. a service to get current tax rates may configured to live for several days in the cache, while a service to get current exchange rates may use the same cache store, but be configured to expire after an hour. The CacheReceive and CacheReceiveXml pipelines contain properties for configuring the cache – it can be enabled/disabled for all messages, or (for the XML pipeline) only for configured message types:

Message-level configuration uses an SSO application store, which holds a MessageCacheConfigurationCollection object in XML:

<messageType>http://schemas.sixeyed.com/CacheAdapterSample/2009#GetServerDateTime</messageType>

</messageCacheConfiguration>

<messageType>http://schemas.xyz.com/2009#GetXYZ</messageType>

</messageCacheConfiguration>

...

Caching is enabled for a message if:

IsCacheEnabled is true on the pipeline AND
- IsMessageTypeConfigurationEnabled is false on the pipeline OR
- IsMessageTypeConfigurationEnabled is true on the pipeline AND
  - the incoming message type has an entry in the MessageCacheConfigurationCollection in SSO.

The lifespan of the object put into cache will be either the cacheLifespan specified in the MessageCacheConfiguration in SSO, or the defaultLifespan specified in the cache send port:

(The point at which the object is removed from cache will depend on the provider, some will remove it when its lifespan expires, others only when its lifespan expires and the cache is full or the cleanup schedule runs).

Installation

For version 1.0.0.0, the CodePlex project comes with three project releases, which can be used independently or together:

Sixeyed.CacheAdapter.msi - a BizTalk application MSI which can be imported and installed to your BizTalk server(s). The application contains the configuration schema and pipelines, so it can be used as-is, or the relevant artifacts can be moved to your own application;
Sixeyed.CacheAdapter.Binaries.zip – a ZIP file of the adapter binaries you can deploy manually. Use instead of the CacheAdapter MSI, if you don't want to deploy the whole BizTalk application;
Sixeyed.CacheAdapter.CacheViewer.msi – a Windows forms application which lets you view the contents of a cache store (the app uses the configuration components from the adapter management assembly, so it is also cache-provider agnostic):

- a documentation release:

Sixeyed.CacheAdapter.chm – documents the .NET assemblies of the adapter

- and two sample releases which should be deployed together:

Sixeyed.CacheAdapterSample.msi – a BizTalk application MSI which has sample usage of the cache adapter, demonstrating orchestration and messaging approaches;
Sixeyed.CacheAdapterSample.Service.msi – a WCF MSI which provides a service which the BizTalk app can cache.

Pre-requisite for the initial release is the non-commercial NCache Express to use as the cache provider.

To add the sample message-level cache configuration to SSO, the SSO Config Tool can import the provided .ssoconfig file into SSO.

I'll update the project to add a memcached provider, and write an updated post when it's available - and the same when Velocity is released. One advantage in having multiple providers would be in evaluating their performance against your expected caching load, so I'll look at extending the sample solution and adding some LoadGen scripts.

And as Michael Stephenson commented, a nice alternative would be to move the caching up to the WCF layer as a behavior, so WCF requests would check the cache first. Two options for this – it can either be a client behaviour using a local cache store (which could be more performant and would have less impact on server resources, but requires the cache provider to be deployed client-side), or as an operation behaviour using a shared server-side cache (which could be more performant, depending on the type of messages being cached, would impact on server and network resource, but would make the caching transparent to consumers). The complexity is on caching the response message if not already cached – the cache key from the original message will have been lost, so there needs to be a different approach for maintaining state to correlate the WCF service request and response.

Visualizing SOA’s Return on Investment

EltonStoneman — Thu, 26 Mar 2009 18:36:18 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

This is the rather grand title of a WebCast I've recorded recently which tries to illustrate where the return on investment comes after moving to a SOA strategy. The video is on Digital Forum: Visualizing SOA ROI , and the Open Source proof of concept it references is on CodePlex: ESBSimpleSamples.

In the WebCast I start with a blank Visual Studio solution and create a new Web app which consumes an existing service. All in it takes 3 lines of code and around 3 minutes of development (most of it spent correcting typos), and it doesn't require me to have any understanding of the underlying SOA infrastructure – in this case, Microsoft's ESB Guidance package sitting on BizTalk Server 2006 R2. Equally important, assuming the service we consume is in production, the testing and deployment effort needed for the new project is minimal.

This is a rework of a presentation I did with a client where I wanted to show how quick and easy it is to use their SOA implementation. I had reservations of showing a code demonstration to an audience which included IT heads (IT Director and Solution Architects) and project management, but it was surprisingly well received, so I've reproduced it using services and components which are publicly available.

Direct links to the references in the WebCast are here:

ESB Guidance: http://www.codeplex.com/esb
UK SOA/BPM User Group presentation: http://sbug.org.uk/media/p/129.aspx
Blog posts on ESB Guidance: http://geekswithblogs.net/EltonStoneman/category/7947.aspx

Using WCF-Custom Bindings in Dynamic Ports

EltonStoneman — Fri, 06 Mar 2009 23:38:49 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

BizTalk 2006 R2 ships with WCF adapters and pre-configured settings for common bindings - basicHttp and wsHttp being typically used for SOAP messaging. With a static port you can use the WCF-Custom adapter, select an existing binding and configure it further in the UI, with the full set of binding options available to you:

Here I'm using basicHttp, but I've configured the maxReceivedMessageSize, sendTimeout and transferMode settings to allow us to call long-running WCF services which return large responses.

In a dynamic port, you're limited to the settings you can configure, as the WCF Adapter Property Schema doesn't contain the full set of binding properties. You can set the sendTimeout and maxReceivedMessageSize on an outgoing message using code:

port(Microsoft.XLANGs.BaseTypes.TransportType) = "WCF-BasicHttp";

requestMessage(WCF.MaxReceivedMessageSize) = 104857600;

requestMessage(WCF.SendTimeout) = "00:10:00";

- but there's no way to access less common properties like transferMode if you're using this approach. Originally this is how we were configuring our outgoing messages, using SSO to store the values used for the binding properties (see Receiving large WCF response messages in ESB Guidance), but we soon ran into an issue with one of the BizTalk servers running out of memory when attempting to process a large WCF response message:

System.InsufficientMemoryException: Failed to allocate a managed memory buffer of 104857600 bytes. The amount of available memory may be low. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

at System.ServiceModel.Diagnostics.Utility.AllocateByteArray(Int32 size)

For basicHttp, the transferMode setting allows you to specify that messages should be streamed (one way or both ways), or buffered - the default is buffered, so when we receive the response message, although it will be streamed in the BizTalk stack, it is completely loaded into memory by the WCF stack. Note that WCF is trying to allocate the full value specified in maxReceivedMessageSize – 100 Mb – even though we've set this as a practical maximum, and the actual incoming message was half that size.

To remedy it, I've switched to using the WCF-Custom transport, and specifying basicHttp with my additional settings in the BindingConfiguration property:

port(Microsoft.XLANGs.BaseTypes.TransportType) = "WCF-Custom";

requestMessage(WCF.BindingType) = "basicHttpBinding";

requestMessage(WCF.BindingConfiguration) = "<binding name=\"basicHttpBinding\" sendTimeout=\"00:10:00\" maxReceivedMessageSize=\"104857600\" />";

The BindingType needs to be specified in addition to the configuration. For the BindingConfiguration value, the Properties page for a static WCF-Custom port allows you to export the settings, so you can configure it in the UI and then save the XML representation, rather than coding it all by hand.

By shifting the BindingConfiguration value to the SSO config store, we'll have the full range of WCF configuration available to change at run-time, so a switch to using wsHttp or netTcp bindings is just a setting change using the SSO Config Tool, and a change to the endpoint address (which we configure in UDDI in this case). Originally I wanted finer control over what could be configured, so we could limit changes to a known set of properties. But having tried this approach I prefer it – it means switching bindings and adding or changing any settings can easily be done without modifying the solution – and I'll be recommending it as best practice.

MSDTC Issue with Clustered BizTalk Host

EltonStoneman — Mon, 23 Feb 2009 23:48:55 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

We had an issue with one of our BizTalk estates with incoming messages being suspended if they were bigger than the large message threshold:

A response message sent to adapter "WCF-BasicHttp" on Receive Location: "x" with URI:"y" is suspended. Error details: There was a failure executing the response(receive) pipeline: "z" Source: "Unknown " Send Port: "x" URI: "y" Reason: 0x8004d027

The error message wasn't particularly helpful but the reason code 0x8004d027 had a few threads and posts relating to DTC. This made sense as we're in a distributed environment so DTC will be used to create message fragments under a transaction, if the whole message exceeds the threshold size. Using Web Service Studio I found the expected message size, and setting the large message threshold above that figure fixed the problem so that seemed to confirm it.

We finally tracked the issue down to permissions on the cluster where MSDTC was running. Buried in the Event Log for the BizTalk host suspending the messages was this error:

The application could not connect to MSDTC because of insufficient permissions. Please make sure that the identity under which the application is running has permission to access the cluster. Please refer to MSCS documentation on how to grant permissions. Error Specifics: d:\nt\com\complus\dtc\dtc\msdtcprx\src\dtcinit.cpp:652, Pid: 6772 No Callstack, CmdLine: "C:\Program Files\Microsoft BizTalk Server 2006\BTSNTSvc.exe" -group "BizTalk Group" -name "x" -btsapp "y"

This was logged when the BizTalk service started, which had been the day before we found the suspended messages, so I didn't see the connection. This post from Ben Cops: http://bencops.blogspot.com/2008/12/error-on-installing-sso-on-clustered.html pointed me in the right direction, and the steps for adding permissions are simple - the only oddity is that you set permissions at the cluster level, not at the resource level:

Open Cluster Administrator
Right-click the cluster (the topmost node) and select Properties
Under Security, add the BizTalk user group with Full Control

This and a host restart fixed the issue. Along the way we did notice a couple of other issues in our MSDTC configuration which we tracked down and fixed with DTCPing. Worth noting that in Windows Server 2003 the properties for MSDTC under dcomcnfg may not correctly reflect the registry settings, so while you think DTC security is configured like this:

- it may actually be configured like this:

The DTCPing log will tell you what registry settings are in use, and the BizTalk Best Practices Analyzer will warn you if the DTC security settings are incorrect, so worth double-checking this.

An Infrastructure Design for ESB Guidance

EltonStoneman — Mon, 02 Feb 2009 14:08:46 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

Our ESB Guidance project is nearing code-completion and we're running System Test on an infrastructure which is suitable for production use. Designed for performance, reliability and scalability, we use servers in a combination of NLB farms, Windows clusters and SQL clusters. Shown below is a modified version of the layout, focusing on the BizTalk landscape:

A few of the design decisions are worth looking at:

Coordination Services run a host for Scheduled Task and FTP adapters, building itinerary requests from received messages. Both adapters are unable to flag an incoming message as being in-process, so are not safe on multiple instances – we have two servers in an active/passive Windows cluster to provide redundancy;
Itinerary Services run a host for incoming itinerary requests from the SOAP adapter. This is instance-safe, so we have two servers configured in a farm for redundancy and performance;
Itinerary Processing runs a host for the custom orchestration we have for handling calls to WCF service providers (not shown in this diagram, but running on a separate Web farm and SQL instance). All BizTalk servers run instances of this host, so any with capacity can service calls;
Enterprise Single Sign-On is located it on the Windows cluster with the Coordination Services, as it cannot run under an NLB farm. The Coordination Service is expected to have spare capacity, so hosting SSO shouldn't impact performance;
UDDI Services run on the BizTalk NLB farm with the Itinerary Services. A small compromise here from the ideal of having a separate UDDI host, but resolver calls will all be made during Itinerary Processing and we use a custom UDDI resolver which holds the cache indefinitely (until manually invalidated when a change is promoted to the UDDI Service), so after the initial warm-up, UDDI Services will have negligible usage;
UDDI Database runs on the same SQL instance as the BizTalk databases. The second small compromise – breaking the cardinal rule that BizTalk should have a dedicated SQL instance with dedicated hardware… As UDDI calls will be minimal, this is a worthwhile performance trade to get the redundancy of having UDDI on the SQL cluster.

The design is intended to cope with a relatively low initial throughput – 250,000 messages in the first year, 98% of which will be a few kilobytes, 2% running into megabytes. BizTalk servers all running Enterprise Edition in a Group, with just Microsoft.Practices.ESB and our custom application installed. Currently the BizTalk servers are running single quad-core CPUs with 4Gb RAM.

Our next task is to run some simulated load through the setup and modify the host layout or physical capacity if necessary. I'll cover the load testing results in a future post.

BizTalk Deployments with PowerShell

EltonStoneman — Thu, 11 Dec 2008 05:09:56 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

As part of my first look at ESB Guidance 2.0 (which has some excellent enhancements which I'll post about shortly) I've been interested in the way the sample solutions are deployed. The P&P team have put together PowerShell scripts which take care of the usual things you'd find in a deployment scenario – stopping services, creating applications, adding resources and bindings.

In a Continuous Integration environment, you'd typically want to deploy to a BizTalk instance as part of the build process, to ensure the application configuration is valid, and as a prerequisite for running unit and integration tests. Currently I do this with MSBuild scripts using a variety of community, Sdc and custom tasks, and command-line tools. It makes for fragmented scripts that are difficult to read and maintain, and can be fragile to execute.

In the PowerShell version a central script isolates the complexity for common functions, and then the individual install scripts use a neat, readable and hugely intuitive syntax. This is a complete install script for one of the sample solutions (MultipleWebServices_install.ps1), which builds and then deploys the artifacts:

$relativePath = "..\.."

$samplesPath = $solutionPath+"\Samples\MultipleWebServices"

. ..\..\..\..\Core\Install\Scripts\ESBFunctions.ps1

function BuildItinerarySolution

{

param([System.String] $buildType)

&$FrameworkPath"\v3.5\MSBuild.exe" $relativePath"\GlobalBank.ESB.Samples.MultipleWebServices.sln" /t:Rebuild /p:Configuration=$buildType

}

BuildItinerarySolution Debug

StopBTSApplication "Microsoft.Practices.ESB"

StopBTSApplication "GlobalBank.ESB"

StopBTSServices

ImportBTSResource "GlobalBank.ESB" "BizTalkAssembly" $relativePath"\Source\ESB.MultipleWebServices.Maps\bin\Debug\ESB.MultipleWebServices.Maps.dll"

ImportBTSResource "GlobalBank.ESB" "BizTalkAssembly" $relativePath"\Source\ESB.MultipleWebServices.Orchestrations\bin\Debug\ESB.MultipleWebServices.Orchestrations.dll"

GacOperation "-if" $relativePath"\Source\ESB.MultipleWebServices.Maps\bin\Debug\ESB.MultipleWebServices.Maps.dll"

GacOperation "-if" $relativePath"\Source\ESB.MultipleWebServices.Orchestrations\bin\Debug\ESB.MultipleWebServices.Orchestrations.dll"

ImportBinding "GlobalBank.ESB" $relativePath\Install\Binding\GlobalBank.ESB.MultipleWebServices_Bindings.xml

StartBTSApplication "Microsoft.Practices.ESB"

StartBTSApplication "GlobalBank.ESB"

StartBTSServices

The common script (ESBFunctions.ps1) defines the main functions – ImportBTSResource etc. – so the logic in the individual scripts is very clean. Common functions make use of PowerShell commands, the BizTalk Explorer OM, BTSTask and custom tools as appropriate, so there's still a variety of tools under the covers, but the common script is a facade over them all. This makes the install scripts neater and abstracts them from the actual implementation of the functions. Below are snippets for the functions to stop and to remove a BizTalk application:

function StopBTSApplication

{

param([string]$appName)

$exp = New-Object Microsoft.BizTalk.ExplorerOM.BtsCatalogExplorer

$exp.ConnectionString = $btsConnectionString

$app = $exp.Applications[$appName]

if($app -eq $null)

{

Write-Host "Application " $appName " not found" -fore Red

}

else

{

if($app.Status -ne 2)

{

#full stop of application

$null = $app.Stop(63)

$null = $exp.SaveChanges()

Write-Host "Stopped application: " $appName

}

function RemoveBTSApplication

{

param([string]$appName)

Write-Output "Removing BTS Application "+ $appName

BTSTask.exe RemoveApp /A:$appName

}

There's some complexity in these common functions, but they're all tried and tested by the Patterns & Practices team. This is a new approach for the ESB Guidance installs (v1.0 used a combination of batch files to deploy the samples), but I don't think there are any inherent dependencies on BTS 2009 so this approach and the common script should work for BTS 2006 solutions. I'm going to look at migrating my current build scripts to this approach as it's so much simpler to work with, and the execution doesn't rely on chaining together unrelated tasks in MSBuild.

The ESBG samples specifically name resources to deploy, whereas my current scripts run dynamically over whichever artifacts they find, but Bart De Smet has a nice MSBuild task for running PowerShell scripts, which takes parameters so that's one option, and generating PS scripts using the ExecuteT4Template task is another.

If I do get this running for BizTalk 2006 R2 solutions, I'll post a working set of sample scripts.

Introducing: BizTalkCop

EltonStoneman — Fri, 14 Nov 2008 16:36:06 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

To make good on a rash comment around the lack of standards-enforcement for BizTalk, I've put together a set of FxCop rules which allow you to run static analysis over BizTalk solutions. It covers all aspects by running rules against compiled artifact assemblies and configured BizTalk applications. It's on CodePlex here: BizTalkCop.

The first release(1.0) contains a full ruleset for validating BizTalk application artifact names, based on Scott Colestock's naming conventions. By default the rules are more strict in places, but they're configurable so you can modify them to suit your own standards.

Usage

Currently BizTalk projects don't integrate fully with static analysis in VS 2005, so you'll need the separate FxCop UI – latest version is FxCop 1.36. Once you've installed FxCop, run the BizTalkCop installer and it will add BizTalkCop.Rules.dll and BizTalkCop.Rules.chm to the FxCop rules folder (default: Program Files\Microsoft FxCop 1.36\Rules). Open FxCop and there are three steps to configure your BizTalk project:

1. Add Targets

Add the compiled BizTalk assemblies you want to check to the FxCop project. For compiled artifacts, BizTalkCop covers orchestrations, maps, schemas and pipelines so you can add any assemblies that contain these:

2. Select Rules

If you've used the default installation, then the BizTalkCop rule set should be listed in the Rules tab (if not you'll need to navigate to add it). Check BizTalkCop and uncheck all the others:

3. Set up Application Configuration

In order to validate the configured BizTalk Application, you'll need to specify the location of your management database, and the name of the Applications to check. Open the ApplicationConfig rule properties (this is a dummy rule which is only there to capture BizTalk Server config), and modify the Settings to specify your server, and the Applications to run over (semi-colon separated if more than one) – you only need to do this once, and all Application rules will use these settings:

When you run FxCop it'll list all the rule failures, some of which may have multiple issues (e.g. ReceiveLocationNamesShouldIndicateTransport will flag an issue for every configured location that doesn't have its transport type as a suffix). There should be no locks on targets, so you can change the artifacts and recompile, or edit application config with FxCop open, and then re-run it.

Most rules are configurable, so you can modify the prefixes and suffixes to suit your standards, or ignore any rules you don't care for. Save the project as a .FxCop file and you can integrate this into your build process using the MSBuild Community Tasks.

Development

There's potential for rules to cover a lot more than the "Naming" category. Certain of the recommendations in the BizTalk Performance Optimization Guide are candidates for a "Performance" category, and there are a few gotchas that could be avoided with warnings from a "Design" category. I'm not planning to add these for a while, but if you want to get involved let me know and I'll add you as a contributor.

FxCop runs over compiled assemblies, so introducing it to BizTalk meant for a few workarounds. If you want to extend BizTalkCop by adding new rules, there are a set of base classes which will help:

RuleBase – general base class for rules using normal introspection over classes; contains methods for determining if targets are maps, schemas etc;
ConfiguredRuleBase – base class for rules which have default setting values which can be overridden in the project file;
ApplicationRuleBase – base class for rules analysing BizTalk Applications; wraps the Explorer OM (so not good for 64-bit) to get the current config;
OrchestrationRuleBase – base class for analysing Orchestrations; typically the compiled code is too dense for analysis, so the base class exposes methods for accessing the shape and OM configuration. This is stored as XML in the compiled class, and the base class provides XPath helpers to access it.

The code for running the rule is quite straightforward, although tracking down the XML elements to check can be fiddly for orchestration rules (if you can find any schemas for the shape or OM XML, let me know and I'll modify it all to use entity classes). The source code contains a sample BizTalk project which fails all known rules, so you can add to this to check your own rules run as expected.

Quite possibly the CSD have something like this in place for the VS 2008 integration coming in BizTalk Server 2009. If not, it should be a trivial job to migrate these rules to run under VS, and in any case the ability to separate them out for running in the build should be useful.

I've run the rules over a few solutions and they've been accurate, but if you come across any issues, log them on CodePlex and I'll have a look.

ESB Guidance Demonstration: “TriathlonResults”

EltonStoneman — Sat, 25 Oct 2008 12:10:16 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

At the SBUG meeting last week, my session on "ESB Guidance: A Real-World Implementation" was meant to end with a demonstration, but we ran out of time - and in any case what I'd intended to show was probably a bit ambitious, with plenty of this-is-bound-to-go-wrong points.

But I've spent some time making it more solid and the code is available now on MSDN Code Gallery: ESB Guidance 1.0 Demonstration: TriathlonResults, if you want to have a look. You'll need an environment with BizTalk 2006 R2 and ESB Guidance 1.0 set up to run it, but I'm looking at putting together a Webcast to give a walkthrough.

Overview

The demonstration involves a set of systems which each have custom integrations to a central system, and shows two SOA approaches for integrating them with BizTalk and ESB Guidance. The first approach is a partial migration, where the client systems are unchanged, but the central system picks up messages directed through ESBG. The second is a "full" SOA migration, where the clients are changed to integrate directly with ESBG.

For the sake of an interesting demo, the systems are contrived to be unusual. They represent different systems for recording an athlete's time in a sector of a triathlon – each having a different UI and a different integration approach. The first sector - the swim - is recorded from a console app:

The second – the bike – from a WinForms app:

And the third – the run – from an Excel 2003 workbook with some VBA script:

The integrations add results to a central database, which has a particularly poor web front-end:

Pre-requisites

Windows Server 2003 with IIS & UDDI Services installed; SQL Server 2005; BizTalk Server 2006 R2; ESB Guidance 1.0.

The sample expects everything to live on localhost, and uses a SQL Server login so you need to have mixed authentication and named pipes enabled in your SQL Server instance.

Installation

Two steps after downloading the code – build and deploy the TriathlonResults.Phase1 solution (you'll need to modify the BizTalk projects to point to your own management database), then run the Deploy.cmd script (it'll ask for the name of your SQL Server – just use "." for a local un-named instance).

This will create:

two SQL databases – TriathlonResults and TriathlonStaging;
a SQL login – TriathlonResults;
a BizTalk application - TriathlonResults;
two IIS virtual directories – TriathlonResults.Web and TriathlonResults.Central.

In the Shortcuts folder under the solution root you'll have links to the various client apps, and to the website which shows the full results from the central system.

Phase 0

This is the starting point, with each system integrating to the central results service in a custom way:

Console app: receives user input and outputs as a delimited flat file; a central console app runs a FileSystemWatcher over the expected location, parses the flat file and inserts the record to the central database;
WinForms app: receives user input and sends the sector result as a Web service call to the central service, which inserts the record to the central database;
Excel workbook: receives user input and inserts the result as a record in a staging database; a trigger on the staging table retrieves the new record and inserts it to the central database.

To see this, run SetupPhase0.cmd and input some results. Usage of the client apps should be fairly straightforward – choose Athlete id=1 and Race id=1 to use known reference data, and refresh the web page for the first race after each input. You should see the sector times displayed after you enter them.

Phase 1

This is the first SOA approach, showing an option for partial migration. Here the central result-recording Web service has been exposed as a service through ESBG, and the clients all use the central service. The WinForms client has been changed to invoke the ESBG call directly, but the other systems are unchanged and a BizTalk app picks up their output and submits it to the ESB:

Console app: flat file output is now monitored by a BizTalk FILE Receive Location. The pipeline for the location parses the flat file into a known schema, and adds the itinerary details as context properties;
WinForms app: uses a typed Service Request class to submit messages to ESBG directly. This is a proxy class which deals with wrapping the message in an ESBG itinerary, and submitting it to the on-ramp;
Excel workbook: the trigger on the staging database is disabled and replaced with a SQL Receive Location which polls the staging table for unprocessed records. The pipeline adds itinerary details to the context;
TriathlonResults BizTalk app: for the FILE and SQL Receive Locations, the BizTalk app has custom orchestrations which map incoming messages to the contract for the Web service, and submit the request. The send port DynamicRequestResponse subscribes to any pre-formed itinerary message, so this picks up the message from the WinForms app, which doesn't require any specific workflow.

To use, close down any client apps you have running, run SetupPhase1.cmd, open the clients and input some results. The command clears down the results DB, so you should start with an empty results page on the website. As you use the clients you can confirm they're using ESBG by checking HAT. You'll probably also notice that the WinForms app responds more slowly when you submit a result.

Phase 2

This is a full SOA migration, where the consumers are aware of the mechanism for requesting services, unlike the Phase 1 approach where this was abstracted away from the legacy apps. All client apps are changed here to invoke the ESBG service request directly:

Console app: now uses the typed Service Request used by the WinForms app to build the ESBG message and submit it. Flat file no longer generated;
WinForms app: unchanged from Phase 1;
Excel Workbook: the typed Service Request is registered as a COM-callable Type Library and referenced from the Workbook. The VBA instantiates, populates and submits the ESBG message. Staging database no longer used;
TriathlonResults BizTalk app: the SQL and FILE receive locations are no longer used, all incoming messages are of the expected itinerary type and are all dealt with by the DynamicRequestResponse port.

To use, close down any client apps you have running, run SetupPhase2.cmd, open the clients and input some results. Again, you should start with an empty results page on the website and can use HAT to confirm messages are going through BizTalk. You'll notice now that all the apps are responding more slowly when you submit a result.

Notes

The Phase 1 approach is a pragmatic one, realising some of the benefits expected in SOA at the cost of others. We have reuse of loosely-coupled services, a centralised discoverable service repository, and centralised message flow. We don't have a common communication mechanism and we haven't got away from bespoke formats and protocols. In some environments this may be as far as the migration goes, if service consumers can't be modified. It has the benefit of being transparent to consumers, but with bespoke mapping and configuration for each consumer it will have the largest custom codebase – taking more effort to develop and maintain.

Phase 2 gives us the benefits of Phase 1, plus we are now using a single entry point to the ESB and consumers are submitting messages of known types which can be verified against the service contract schemas held in BizTalk. This particular implementation sees all the apps using a .NET component to abstract the work of creating the itinerary. This means consumers need no knowledge of BizTalk or ESB Guidance, but has the complication of distributing and maintaining that assembly. Consumers could use any approach to build their requests, provided they can understand XML and SOAP.