Michael Stephenson

BRE Survey

Michael Stephenson — Sat, 25 Oct 2008 22:32:46 GMT

Apparently the CSD/BizTalk Product Group are conducting a survey on the BRE to analyse its usage. I guess this is an opportunity to contribure to the future of this component so it would be useful for BizTalk and probably WF people to take this

https://live.datstat.com/MSCSD-Collector/Survey.ashx?Name=BRE_Usage_Survey_Blog

MVP

Michael Stephenson — Thu, 02 Oct 2008 21:09:41 GMT

Ive just been awarded a Microsoft MVP for BizTalk.

From what the other guys have said about it im really looking forward to the opportunity to be involved in interesting stuff over the next year

NCache and BizTalk Cross Referencing Example

Michael Stephenson — Sun, 21 Sep 2008 18:48:13 GMT

Article Source: http://geekswithblogs.net/michaelstephenson

Following a recent post about the different approaches to caching you might consider when implementing reference data mapping in BizTalk one of the things that stands out most was that the solutions where a team had used a caching approach often resulted in them not using the BizTalk Cross Referencing features. As I've mentioned many times I prefer to use this unless there is good cause not to (there are reasons where you might not want to) but I feel development teams often ignore or don't consider the impact of adding custom databases to a solution without consideration for the extra work this requires in development, testing , deployment and management.

In most cases why do you want to do this when you already have a data store designed for this purpose? One criticism I would make of BizTalk is that the product does not do a very good job of making it easy for people to use the cross referencing featured from a developer experience but these can all be worked around with few problems.

Anyway I have decided that I will produce this sample showing how I have combined the use of NCache and BizTalk Cross Referencing to get a solution which does not need custom databases yet can still have a high performance caching solution which will not increase the BizTalk hosts process memory unnecessarily. The sample can be downloaded from the bottom of the article.

Prerequisites for the sample

You can obviously review the code in this sample, but if you want to run it you will need to do the following things:

Install NCache Express Edition

NCache Express Edition is available free from the following link http://www.alachisoft.com/ncache/. I assume you will be installing this to the default location, but if not you might need to modify the msbuild script where I configure NCache.

Modify Cross Referencing Setup

In the SetupFiles.xml file in the solution it contains the xml used to setup the cross referencing data in BizTalk. This file requires absolute paths to work so you will need to tweak these to suit your location. The below picture shows what the xml looks like.

Setting up the sample

In the sample you will notice there is a file which is called Setup.cmd. If you run this file it will perform the appropriate actions to configure things for this sample. The actions it will take are as follows:

Stop the cache in NCache if it is already running
Clear the BizTalk cross referencing tables
Stop the NCache windows service
Copy the pre-configured NCache config files to their appropriate places to configure NCache with the cache we will use in this sample
Start the NCache windows service
Start the custom cache
Load the BizTalk cross referencing tables using BTSXRefImport

These actions are all done in an msbuild script (picture below) which should make it easy for you to see how this is done.

You should now be able to run the sample.

My Cross Referencing Component

To keep the sample simple I have developed a component which will provide an interface which is the same as that provided by BizTalk cross referencing. I provide a class called CrossReferencingFacade which implements the façade pattern to give you an easy way to obtain the common and application specific id's. The below picture shows this:

There is also a test in the test project which shows how to consume this component. It is as easy to consume as the BizTalk cross referencing dll. If you look in the CrossReferencingManager class you will see there are two key methods which are discussed below:

LoadXRefIDData

This will use some data access code to retrieve all of the cross reference data for one specific type of cross reference (xrefId) for example all of the mappings for Product Type. It will then return them to the calling method.

GetCrossReferenceIDData

This method will check the cache to see if the data is already there for the requested cross reference data type. If present it will be returned from the cache, and if not the data will be loaded using the LoadXRefIDData, and then placed in the cache.

The result of this means the data is cached once for both the GetAppID and Get CommonID methods.

One interesting bit on this (and there may be better ways to do this) is that to allow you to search for the appropriate mapping data from the same source by both CommonID and AppId I have held the data in a container object which houses a dictionary of the reference data with a unique key for each one, and then I also have 2 dictionaries of the app specific keys and common id keys. This just makes it possible to hold the data just once but search for it in different ways. As mentioned I'm sure if I have a think about this there are better ways to do this but it will do for this sample. (note although this last bit may have sounded overly complicated this is encapsulated so the consumer does not need to care about this)

The NCache bit

So from the above hopefully you can see I have provided a handy way to use BizTalk cross referencing within this sample. The next thing to discuss is NCache. I believe there are a number of additional features which come with the Enterprise version such as security features and tools to manage caches so for any production usage I would definitely recommend that version. For the purposes of this sample the Express edition is more than sufficient.

You can see from the below picture that the code to interact with NCache is very simple.

(Note: In the above picture you can only partly see it, but the cache allows you to insert objects with expiration parameters and also a cache dependency)

With NCache one of the things that I like is I don't need to worry about configuration within my application, so you will notice there isn't any app.config files in the sample which are used by the consumers of the cache. That said there is some configuration for the caching service. You will see in the NCache folder there is some config which controls how your caches are setup. The below picture from the config.ncconfig file shows how I have configured my cache for this sample.

You will notice here that I'm able to control if my cache runs in process or out of process which is how I'm able to move the cached data outside of my Biztalk host process and there are a bunch of other possible settings. This configuration is held along side the caching service.

Plugging it into BizTalk

This component is now very easy to add to a BizTalk implementation by using the call external assembly feature of the scripting functoid to call the component. You will now be able to use the cross referenced features of BizTalk but with out of process caching of the data.

Summary

Hopefully you will see that it is not that difficult to implement a good caching solution which has addresses a combination of the considerations I discussed in my previous article. I quite like this approach and based on my limited experience of the different caching systems available I would probably at present choose NCache over Memcached because it is an established 3^rd party system which comes with additional tools and features to support it. That said I will be keeping an eye on the "velocity" project as I think this will definitely be one to watch for the future.

If you have any experiences with this I would be interested to hear your thoughts. The sample is available below:

Caching Strategies for Reference Data Mapping with BizTalk

Michael Stephenson — Sun, 21 Sep 2008 18:42:25 GMT

Article Source: http://geekswithblogs.net/michaelstephenson

I've been asked the same question a few times recently by a couple of BizTalk projects about how to map their reference data. When this question comes up we often get involved in a discussion about the pros and cons of caching the reference data and increasing memory usage versus hitting the database every time.

As a rule I tend to use the BizTalk Cross Referencing features for this data mapping unless there is a specific requirement which requires some custom approach. I've blogged about this kind of thing a few times before but I thought its worth a post with some thoughts on the different approaches I've seen used when people have wanted to use caching.

I mentioned in a previous post that the Value cross referencing features already implement a simple caching mechanism. In my opinion though the value cross referencing is aimed more at mapping data type values between types of systems rather than business reference data which would be held in instances of systems which is what I feel the ID cross referencing is aimed more at.

Anyway when it comes to this design decision the things people are usually trying to balance are as follows:

Performance – If I have a lot of things to map I don't want to be hitting the database thousands of times
Performance – If a I cache the reference data is there a risk it will consume a fair bit of memory and potentially cause throttling based on host process memory threshold being hit
Managability – If I cache the data it will have an instance of the cache in each host instance that uses it. How will I ensure these stay in sync
Managability – Caching will mean I have to restart all the hosts when the data changes

There are a number of possible ways to solve this problem and each have their own considerations which are discussed in the rest of this article.

Simple Singleton Approach

This is probably the most common approach I've seen. In this approach I've normally seen a custom database implemented to manage the reference data. The developer would then implement a custom data access method and a singleton which would be used to control access to the reference data. This is a pretty standard use of the singleton pattern. In this approach I think some of the considerations which need to be made are:

Pro's

Fast access to the data
Easy to implement in terms of the C# coding

Con's

In most cases there is additional development of a database to manage the data. This then involved additional development/testing/deployment and management work
The data is cached in the host process so you need to watch for the impact on the process memory of the BizTalk host
If you access this reference data from BizTalk maps running in different hosts then you may end up with multiple instances of the cached data on each server
By default your cache usually will not detect changes to the underlying data, however with additional coding you can monitor the custom data and update any changes
In most cases the hosts need to be restarted to pick up changes
The cache will not be cleared when the data is no longer used

Caching the Response from a Web Service

Sometimes I've seen an approach where a custom database has been implemented then a web service façade has been implemented on top of it. The web service will access the data and return it. In consuming this from BizTalk a C# assembly has been developed which uses the web service to get the reference data which is then consumed by a map.

Pro's

The caching is outside of the BizTalk process
The caching can be relatively easily configured
If the web service is located on the BizTalk box then a local machine hop would be quicker than going remotely, and also with WCF you could optimise this further using appropriate channels

Con's

There is a lot of additional development in this approach, custom coding of the web service, development of the database
There is a lot of additional management and deployment effort for the database and virtual directories etc for the web service

Using the HTTPCache

In this approach I've normally seen it implemented in the same way as the singleton approach above. The key difference is that the reference data is usually held locally in a static hash table in the singleton approach where as in this approach the HttpCache object from the System.Web namespace is used. This gives a couple of options around a sliding and absolute expiration which will remove unused data from the cache helping to control the memory usage. You can also add one of the .net cache dependency objects which would allow you a way to detect changed and refresh the cache.

Pro's

Would be fast access to data
Relatively simple to implement ways to detect changes
Ability to clear the cache for unused data

Con's

Again usually has a custom database for the reference data
This is in process caching so you need to be aware of the memory usage

Using Enterprise Library/Caching Application Block

Enterprise Library has a caching block which provides a number of features which could help you solve this problem. One of the key benefits of enterprise library is that it supports different types of stores for the cached data including:

Null – means just stored in memory
Database
Isolated Storage

If I remember right the cache supports the same features as the HTTPCache approach which allows you to have a dependency and also expirations. There is an article at the following location which discusses using Enterprise Library Caching in BizTalk http://www.malgreve.net/2007/07/using-enterprise-library-in-biztalk.html.

Enterprise Library can also integrate with external backing stores to support out of process caching.

Pro's

Ability to abstract the caching store from the consuming code
Standard caching feature set

Con's

Again usually some requirement for custom data store for reference data
Enterprise Library usually required lots of configuration to setup and manage
Most commonly cached in process so near to be aware of memory usage

Out of Process Caching

One approach I quite like involves caching the data outside of the BizTalk process. This provides the benefit that you can cache without having to worry about the impact on the BizTalk process memory usage. There are a number of caching tools which you can use to help here such as:

NCache – http://www.alachisoft.com/ncache/

Alachisoft offer an express version of their caching product which is free and a version for a relatively small cost which comes with some management tools for their distributed caching system.

Memcached – http://www.danga.com/memcached/

Memcached is an open source distributed caching system. I know of some guys who have used this very successfully on a .net project with a major UK company.

Velocity - http://www.microsoft.com/downloads/details.aspx?FamilyId=B24C3708-EEFF-4055-A867-19B5851E7CD2&displaylang=en

Velocity is an initiative at Microsoft at present to create a distributed in memory caching platform. I feel that as this evolves it is important to keep an eye on this as it will in the future be likely to become the best approach to this.

These distributed caching systems offer the benefit of taking the memory usage out of your process, but offer fast access to the data via their API. Most of these products also offer high availability and synchronisation across a group of caches when you distribute them across your server group. I have in particular looked at NCache for this example and it is setup as a windows service which you would deploy on each BizTalk box. These services would then be configured to work as a cluster meaning they would synchronise themselves when changes were made.

Pro's

Out of process caching offers still fast access to the cached data, but removes the likely hood that the cache might affect BizTalk performance
These caches are designed for high performance such as NCache which is intended for high performance customer facing ASP.net applications
They can be integrated with caching frameworks such as Enterprise Library (NCache comes with this out of the box)
NCache supports cache dependencies and expiration

Con's

Again requires work and management of additional components. I think NCache (the buy version) offers a better management and operations
Potentially brings up the 3^rd party or open source debate around which cache system to use

Summary

Hopefully this article has highlighted the many options available when you are considering a caching solution to support your BizTalk implementation. There are many considerations which can be made and there isn't always a one size fits all rule like in most design decisions. I think some of the things that stand out from this discussion are that most of the approaches above always end up using a custom database to manage the reference data. I think in a future post I will look at how to combine some of the approaches discussed here with the BizTalk Cross Referencing features to produce a fairly simple yet effective combination of all of the approaches.

Basic Server Diff Tool

Michael Stephenson — Sat, 20 Sep 2008 20:14:20 GMT

We had a few problems on a project recently where the server builds did not set machines up exactly the same. This is a pretty major problem as if the software installed across your BizTalk Group you can not expect to receive consistent or expected results. Based on this I reviewed a few servers using PsInfo recently but being a manual thing its a bit of a pain trying to identify any differences.

Im sure there must be tools to do this but a quick google didnt throw up too much joy so I knocked up a quick tool to help.

The tool will let you list a bunch of servers which you expect to be the same and will then run PsInfo agains them and report any differences which are identified.

There are a couple of things to note here though that it will not do things like compare registry settings etc, and ive not used the switch to get disc information from psinfo as if the servers have been used then their disc free space is likely to differ.

Anyway its a bit basic but saved me some hassle so ive shared it on codeplex; http://www.codeplex.com/ServerDiff

Schema problem with source control

Michael Stephenson — Thu, 18 Sep 2008 19:04:36 GMT

I came across a frustrating little problem today. We have a schema for a web service we call and have done for ages. The schema is generated by the WSE adapter wizard.

We have had a couple of issues we have had to troubleshoot and it became apparent it would be useful to track a couple of promoted properties from this schema to help with the diagnosis.

So you would think it's pretty straightforward. Add a property schema and then apply the promotion to the schema. It all works great until you save the schema.

Normally a BizTalk schema is in Unicode encoding, but the WSE adapter generates one with ANSI encoding. When you try to save the schema you get a warning because Visual Studio changes the encoding during the save.

The knock on to this is that we are unfortunately using the worst source control system in the world (not naming any names) which then wont allow you to check in your change because it can not diff the files due to the encoding change.

Bit of a pain in the bum but you can work around this, id be interested to know if there is a reason the adapter does a different format to everything else.

BizTalk XRef Undocumented feature

Michael Stephenson — Sun, 14 Sep 2008 16:35:32 GMT

I found something out about the BizTalk Cross Referencing features the other day that I didnt know and havent seen in the documentation.

If you use the ID cross referencing it will hit the database everytime you use the functoids or access the dll directly

If you use the VALUE cross referencing it implements a simple cache using a singleton so if you update data in the database you should restart each host instance that uses it before the changes will take effect.

Reviewing a BizTalk Group

Michael Stephenson — Sat, 13 Sep 2008 19:59:15 GMT

Article Source: http://geekswithblogs.net/michaelstephenson

I've recently been reviewing some BizTalk setups for various reasons. These include:

Is the setup correct
Performance analysis and issues
General troubleshooting

I thought it would be useful for me and others who might want to look at doing a review of a BizTalk setup to make some notes on some of the activities you might want to do.

Comparing Servers

I've come across a couple of instances previously when servers had been setup incorrectly with missing hot fixes or in one case a missing service pack for the .net framework on one server in a group. When you have a BizTalk group or a clustered SQL Server you want to ensure the servers in the group are all the same. In most cases they should have the same hardware and same software installed on them.

I had a bit of a Google on this and there doesn't seem to be that many tools which easily allow you to do this (if you know of any, or I have missed something obvious let me know), but the main way for doing this is with the PsInfo tool from Microsoft/SysInternals. The key thing with this tool is that I'm looking to validate that all servers have been setup with the correct software and hardware and they are all consistent. PsInfo is available from the following location and there are examples to show you how to do this.

http://technet.microsoft.com/en-us/sysinternals/bb897550.aspx

Update - Ive done a tool to help with this: http://www.codeplex.com/ServerDiff

Reviewing Security with Microsoft Base Line Security Analyser

This is a free tool from Microsoft which will allow you to compare a set of servers against most of the recommended security standards. It will identify any vulnerabilities on the servers you have analysed and highlight them and offer recommendations on how to fix or mitigate the risk associated with it.

I would use this tool on all BizTalk and SQL Servers in the setup. This tool is available from the below link:

http://technet.microsoft.com/en-us/security/cc184924.aspx

Reviewing SQL Server with SQL Server Best Practice Analyser

Most people are probably familiar with the SQL Server BPA, but if not this tool will collect information about your SQL Server instance and then compare this against best practice rules to highlight any issues with your setup. The tool also makes recommendations on how to resolve these issues. This is a very useful tool and is available from the following location:

http://www.microsoft.com/downloads/details.aspx?FamilyId=DA0531E4-E94C-4991-82FA-F0E3FBD05E63&displaylang=en

Reviewing BizTalk Server Group with BizTalk Best Practice Analyser

The BizTalk BPA is a tool which will inspect your BizTalk Group and compare it against well known best practice rules for a BizTalk setup and identify any issues you may have. It also offers recommendations on how to resolve these. The BizTalk BPA is available from the following location:

http://www.microsoft.com/downloads/details.aspx?familyid=dda047e3-408e-48ba-83f9-f397226cd6d4&displaylang=en

Reviewing BizTalk Group with Message Box Viewer

MsgBoxViewer is an excellent tool developed by Jean-Pierre Auconie which allows you to run a whole bunch of queries against your BizTalk Group. It will provide you lots and lots of information about the group and also perform some analysis to advise you of things you might need to review. Rather than get into a lot of detail about it, I would like to refer you to Jean-Pierre's post where he gives you all the information you could want.

This tool is available from the following link: http://blogs.technet.com/jpierauc/pages/msgboxviewer.aspx

Reviewing BizTalk Application setup with the BizTalk Documenter

When your application has been deployed one common problem is incorrect configuration of one of the many settings that are available. By using the BizTalk Documenter you can quickly and easily document your BizTalk system and then review the chm file to easily see what settings are applied. This gives you a change to look over the settings and potentially spot any problems.

The BizTalk Documenter is available from here: http://www.codeplex.com/BizTalkDocumenter

Reviewing Activity by Parsing the IIS Logs

Lots of BizTalk projects will utilise IIS to expose web services to other applications. Using Log Parser to analyse the IIS logs is a good way to get an idea of the activity you are getting in IIS and also identify any errors that are happening. Below is some of the queries I often use to analyse IIS.

Query	Description
IIS Calls by hour	This will give you a breakdown of the number of calls per hour across the chosen day
Calls by hour and URL	This will give you a breakdown of the calls by hour and url across the day
Calls by minute	This allows you to get an analysis of the load on the server in terms of IIS calls per minute
Calls by minute and url	As above but with an additional split by url
IIS Errors	This will list all IIS requests which have responded with an error on that day
IIS Errors by url	As above but summarised by url
Request message size	Lists any large messages which have been sent to IIS (note this required additional IIS logging parameters to be enabled)
Response message size	Lists any large response messages which have been returned through IIS (note this required additional IIS logging parameters to be enabled)
Time taken	Lists the calls in terms of the overall duration of the request (note this required additional IIS logging parameters to be enabled)
Requests per day	Breaks down the calls to count how many there have been for each url for a given day

I have provided a sample of these queries from the samples folder at the bottom of the document.

The RunIISQueries.cmd file shows how to run the queries. You basically need to provide the following parameters

The output path where to place the reports
The path to the IIS log files which can be a remote server
The path to LogParser
The data to inspect for (in the format yyyy-mm-dd)

These queries tend to be useful on a project when you have limited information about expected volumes. We were able to analyse various servers and work out what the current volume information is, and also we used them to occasionally review the usage on test environments. The beauty of Log Parser is that it is very easy to do your own queries so if you use some different queries feel free to mention them in the comments.

Reviewing Activity by parsing the Event Log

The event log is one of the most important resources available to anyone managing or reviewing a BizTalk setup. It will contain information about any problems that have been experienced etc. On one of my projects we have setup some daily reports which will analyse the event log and give us different views on what has been logged daily. This is in addition to any alerts which get created by monitoring the event log with Openview or MOM.

Some of the event log queries we use are as follows:

Description	Query
This query will review the event log and find all events for the BizTalk event source	logparser "Select TO_STRING( TimeGenerated, 'yyyy-MM-dd' ) as Date, TimeGenerated, EventId as ID, SourceName Into <OutputPath>\EventLogBizTalkEvents.txt From \\<ComputerName>\Application Where Date Like '%<Date>%%' And SourceName Like 'BizTalk%%' Order By SourceName, EventId, TimeGenerated" -i:EVT -rtp:-1
This query will review the event log and find all events logged for our custom event source	logparser "Select TO_STRING( TimeGenerated, 'yyyy-MM-dd' ) as Date, TimeGenerated, EventId as ID, SourceName Into <OutputPath>\EventLogBupaEvents.txt From \\<ComputerName>\Application Where Date Like '%<Date>%%' And SourceName Like '<CustomEventSourceName>%%' Order By SourceName, EventId, TimeGenerated" -i:EVT -rtp:-1
This will review the event log and give a count of events for each event source and number	logparser "Select TO_STRING( TimeGenerated, 'yyyy-MM-dd' ) as Date, EventId as ID, Count() as NoEvents, SourceName Into <OutputPath>\EventLogSummary.txt From \\<ComputerName>\Application Where Date Like '%<Date>*%%' Group By Date, ID, SourceName Order By SourceName" -i:EVT -rtp:-1

On the back of this analysis we keep a library of known events in Share point so we have information about all of the events we typically would expect to see. We run these reports and build up our knowledgebase through testing phases and then by the time we move into production we have already got a good ability to support our solution in place.

Quite often too we have come across intermittent problems which are difficult to track down, using Log Parser helps us to analyse the event log for information across multiple servers which might help us to solve this.

Reviewing Activity from HAT with the Orchestration Profiler

The BizTalk orchestration profiler is another useful tool to help you review activity on a BizTalk Group. One of the most common ways I use the tool is integrated into an automated testing process to ensure I have good code coverage. But it can also be used to point at a BizTalk Group and just analyse how the orchestrations are being executed. To be honest it is not really one of the first things I would do when doing a review, but it can help you look for unused orchestration paths and also be useful if you are doing a BizTalk upgrade project and want to see how often orchestrations are used. If I remember right you can also analyse the execution of orchestrations to look for performance bottlenecks within an orchestration.

There is lots of information in the community about this tool so rather than going into it in detail here I would like to refer you to the following site: http://www.codeplex.com/BiztalkOrcProfiler

Reviewing activity from HAT with some custom queries

All BizTalk developers will be familiar with HAT. There are a bunch of out of the box queries which can give you various information and you can create your own queries specific to your solution. This section is more for completeness of the article as there is plenty of documentation about the standard queries, and you will make your own up to suit your own needs.

Performance Analysis with Perfmon and PAL

I'm often surprised how many times I've done a BizTalk interview and asked a candidate about how to analyse a specific performance issue with BizTalk and how few times anyone mentions perfmon. There are loads of performance counters for all of the different features of BizTalk, and you can also create your own to compliment your solution. These counters can give you valuable information about the health of your BizTalk Group.

If you want to perform a very simple performance analysis of your BizTalk Group both Perfmon and PAL can help you do this. The steps would be as follows:

Use Chapter 9 of Darren Jeffords excellent book to review a simple way to setup your perfmon trace for your BizTalk Group using the spreadsheet he provides.
Start your Perfmon trace and run it while your Group is doing its work
Stop the perfmon trace
Run the PAL tool (available here) against the Perfmon output file
Review the PAL report

The PAL tool will review the captured performance information against recognised benchmarks and produce a report which will highlight any problems or things to be reviewed.

Review Configuration Files

One of the other common deployment mistakes comes when you have configuration information held in configuration files such as BTNTSVC.exe.config or possibly held in SSO or some other configuration store. When you have this kind of configuration it often changes depending on which environment you have deployed to and it is a common issue in deployment that the wrong configuration values have been deployed.

I usually give these files a quick sanity check manually, but also to mitigate the risk of this problem happening in the first place I usually use the approach I discuss in my article about configuring binding files to allow me to have a template and way of producing configuration and binding files for different environments by abstracting the values from the binding/configuration file template.

There is also the spreadsheet option available as part of the deployment framework which Scott Colestock created check the following link for details: http://www.traceofthought.net/PermaLink,guid,b9c45d34-85c8-449f-b1a6-deafc2d89084.aspx

Operational Monitoring Tool

Finally if you read my blog regularly you will have seen me going on about how important it is for organisations to monitor their systems with a tool such as MOM/SCOM or HP Openview. These tools will highlight lots of the things which you might find using the above reviewing techniques because they would have preconfigured alerts in them. My two most common questions when reviewing a BizTalk solution related to this are:

Is the customer monitoring their BizTalk environments with something like MOM
Are they staying on top of the alerts that are being raised

If the answer to the above 2 questions is yes then it's a good indicator than an overall review would be good.

Summary

This article was intended to discuss some of the various techniques you might use to perform different types of review of a BizTalk environment. These reviews would aim to ensure that a group has been setup and deployed correctly, and then is managed and operates effectively when it is used.

Id be interested in any thoughts or other things people do in terms of this.

UK BizTalk Events

Michael Stephenson — Fri, 05 Sep 2008 21:32:22 GMT

Microsoft have asked us to advertise the following events to members of the UK SOA/BPM User Group

BizTalk RFID – Connecting the Edge to Enterprise – 29th October 2008

SOA & BPM Vision Briefing – 23rd Sept 2008

SOA/BPM User Group Meeting

Michael Stephenson — Fri, 05 Sep 2008 21:07:25 GMT

The next session of the SOA/BPM User Group has been arranged, we are still finalising content but the event registration is now available.

If you would like to see the details or sign up refer to the following url:

http://sbug.org.uk/forums/p/68/99.aspx#99

Strange Service Window Behaviour

Michael Stephenson — Fri, 05 Sep 2008 21:03:23 GMT

Thought I would add a quick post about some strange behaviour we have been monitoring recently with service windows. Our situation was as follows:

We have 4 ports which monitor FTP locations, 2 of them have service windows and 2 dont. The service windows are specifically for a 1/2 period and have a 5 minute polling period in them. Our servers are not configured for high availability because there are issues with being able to cluster the servers so we monitor the hosts using HP Openview and if they go down a manual intervention is used to initiate a host on the other server. We receive 4 files per day one for each location.

The symptoms we experience are as follows:

The ports which do not have a service window always seem to work fine
The ports with a service window were working fine for a couple of months and then seemed to stop working
If we remove the service window the ports work fine
We can not see any record in the external FTP servers logs that the port has polled for the file on the ports with a service window
We have enabled the logging/diagnostics file for the FTP adapter but nothing is recorded for the service window ports
If you restart the host it will pick up the file if it is within the service window, but the next day the port does not pick the next file up
We have been able to validate from the FTP logs the file was there at the expected time
We have been unable to simulate this on another environment, it only happens on our production environment

Based on the above its a bit strange, we have been able to get it working by swapping the server the host instance runs on, but being our production environment I dont really want to be doing anymore analysis here. It will be interesting to see if this comes up again, and also if we moved the host instance back to the first server if it still breaks

In the analysis I came across Saravana's post about some stuff he had with service windows and I am fully supportive of his suggestion. I think that in addition to that it would be very useful if there was an option where you could enable logging to the event log the start and finish of a service window for a given port. This would be very useful from a monitoring perspective and allow you to configure your monitoring tool to look for these events and work out if a service window had been missed

I hate days like this

Michael Stephenson — Tue, 12 Aug 2008 18:38:08 GMT

Had one of those days yesterday and today, you know what its like when you find a bug/problem and you dont really get a chance to properly look at it when your doing a million things at once.

It is so often that the cause of the problem is really obvious but you just cant see the wood for the trees.

This is a reminder for myself as much as anything.

I was writing an msbuild task to wrap cruise controls ICruiseManager so I can get the latest build label of a project that has been built.

The problem was that when I tested the task using MsTest from Visual Studio it worked fine, but when I ran it from an MsBuild script it kept failing complaining about not being able to find the ThoughtWorks......Remote assembly. I ensured that the assembly was where my tasks assembly way and frustratingly it wasnt working

When my brain returned I spotted that the problem was that the assembly is being searched for in the directory which the process started (the .net framework directory where MsBuild.exe is) so by copying the assembly there it now works.

UK SOA/BPM User Group Slides

Michael Stephenson — Tue, 12 Aug 2008 18:25:11 GMT

All of the slide decks from last months UK SOA/BPM User Group session are now available from our site

http://sbug.org.uk (in the media section)

The decks covered were:

Oslo - Robert Hogg

Composite Applications - Andy James

SOA in 2015 - Darren Hallett

Frends Newsletter

Michael Stephenson — Tue, 12 Aug 2008 18:22:01 GMT

Just been reading the quarterly newsletter from Frends (http://www.frends.com/), it was nice to see a little plug for my recent article about scheduling requirements for BizTalk solutions where I discuss the use of their product as one of the possible ways to implement this kind of requirement:

http://geekswithblogs.net/michaelstephenson/archive/2008/05/16/122203.aspx

BizTalk Documenter in my build

Michael Stephenson — Mon, 04 Aug 2008 23:17:43 GMT

From previous posts you may have read how I integrated the Microsoft BizTalk documenter into my MsBuild process.

In general this has been working fine, but every now and again I kept getting a build failure when trying to generate the documentation as follows:

System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Documents and Settings\<UserAccount>\Local Settings\Temp\BTS2K4Doc\Application\fb7b6ba7-016c-4a1f-9c7d-ec9037911456.html'

If you get this it seems to be caused by an hhc.exe (Microsoft Help Compiler) process still being alive from a previous run. Just kill it and the builds will work again.

Refactoring Tales: Long running splitter pattern

Michael Stephenson — Fri, 25 Jul 2008 21:36:02 GMT

Article Source: http://geekswithblogs.net/michaelstephenson

I have often come across situations where I have been asked to look at a process (usually in BizTalk) where it isn’t quite running as the customer would like. I have decided to start a series of posts which I will call refactoring tales. These posts will discuss a the process implementation and the problems encountered along with it. I will then discuss the approach taken to improve things and what the benefits were.

Background

This particular process had been implemented as a sort of splitter pattern. Each day a file would be received which was then to be split and loaded into two different systems. The split was made by checking each record against a third system which would indicate which system the record should be loaded into.

This example is relatively common and the implementation which was developed was as follows:

1. The file was received as a complex XML file via the FTP adapter which had a service window applied to it A normal file would contain about 5000 records and was approximately 10MB. On exception there have been files as large as 40MB.

2. A pipeline component was applied in the decode stage which would remove a pre-processing instruction from the file

3. The file was delivered to the Messagebox

4. An orchestration would subscribe to the message and start processing

5. Some validation logic is applied to ensure the message is in sequence. The logic involves using the SQL adapter to check a database to ensure the message is in sequence.

a. If the message is not then an event is logged and a manual recovery process is initiated and the orchestration suspended

6. In the orchestration it would iterate through records in the file extracting each record.

7. In each iteration of the loop the extracted record is checked against an external system to see where to send it by using the oracle adapter

8. If the message is sent to system A then a web service call is made using the WSE 2 adapter to load the message

9. If the message is to be sent to system B then an in memory message is built up for all of the messages for system B which will then be sent in batch

10. When all or the records have been processed then the records for system B are send via FTP to that system. The batch message (about half of the original file) is transformed in the send pipeline to a flat file format.

11. The batch management database (a custom database which is very simple and to support the sequence number validation) is updated to mark this batch complete

12. Finally an email is sent to the business users to confirm the results of the batch processing (how many went to each system)

The Pain

The following pain was experienced with this process:

Length of time it runs

Each record in the file would take approximately 4-5 seconds to process with a little time before and after the loop. It was common for the file to take in excess of 6 hours to fully process. The result of this was a file received in the morning would often take almost the whole business day to be loaded.

Cannot scale

One of the key limitations of this design is that because the process is sequential it cannot be scaled out. If the file were to suddenly become 10,000 records you would expect the processing time to double. Adding more BizTalk servers to the group would have no impact on this duration.

Does not recover well from errors

During the first use of this service there were some network connectivity issues and this ment that sometimes the 3 retries on the web service port would all fail. This meant that the exception was caught and the process stopped until someone looked at the problem. When resolved the orchestration was resumed it would continue. But as you can imagine there are two significant impacts on the amount of time the overall process would take.

The first was the response time from a manual operator. This could add anything from a few minutes to an hour that the whole process would be stopped.

The second was that each port error would initiate one of the port retries about 5 minutes later. In the example here about 10% of the calls failed (250+ because not all of the records went to system A) and you can imagine the impact of these extra minutes on the overall processing time. One file took 2.5 days to fully load.

Persistence Points & SQL Transaction Log Backups

One of the behind the scenes problems was that because of the fact that the orchestration was holding one large message in memory from the start and building up another through the execution of the process it meant that every persistence point was expensive with the orchestration state being between 10-20 MB for the standard message size.

There were approximately 2 persistence points per record so for a standard file (5000 records) that means 10000 persistence points. If you do the calculations here you can see we are persisting somewhere in the region of 100Gb worth of data to the messagebox during the processing of this 10 MB file just to save the state of the orchestration.

The SQL Transaction log backups for the messagebox were eating drive space like crazy.

There were a couple of other smaller things too, but as you can see there is a clear business and operational need to refractor this.

There were a couple of lessons to learn from this situation, particularly around performance testing and the difference in how a process will be perceived to run on a developer machine versus how it will run in a live environment, but I will not go into these within this post.

The Refactoring Challenge

Refactoring is different to re-writing. If we were doing this from scratch there would be a number of different things you could do, but one of my favourite things about these kind of situations is there are things you can change and things you can’t. The challenge is to make it work much more effectively while minimizing change/cost. Some of the factors which affected how I could refactor this are as follows:

· No change to external systems

In this case I could not change any of the external message formats or communication mechanisms. There was no opportunity to change the external systems functionality.

· Short timescale

I only had a short amount of time to POC a new idea for this and then it would be implemented by other developers.

· The business users still want their email

The email feature for the users screams out as a BAM opportunity. However in this refactoring there is no opportunity to change this or retrain users on how to use the BAM portal etc.

· Limitations on technologies to use

This particular solution could potentially use SSIS to help with the implementation, in this particular case it was preferred to keep it as much in BizTalk as possible to keep the supportability of it within what had already been trained to the support team.

The New Implementation

After weighing up all of the factors and components I had to work with I decided that the best way to refactor this process would be as follows:

1. The file was received as a complex XML file via the FTP adapter which had a service window applied to it

2. A pipeline component was applied in the decode stage which would remove a pre-processing instruction from the file

3. An XmlDisassembler was added next in the pipeline to promote some metadata about the batch from the message header to the message context

4. A new custom pipeline component was added to the pipeline to rip apart the batch.

This new pipeline component would use the XPathReader to read the message and to extract each record from the message. The database which was part of this solution used to manage batches was extended to have a table to contain a temporary storage for the batch records. The pipeline component would save all of the records to the database and then change the message which would come out of the pipeline component to be a very small trigger message which would be sent to the messagebox and contain the ID of the batch and the number of records it contains.

5. A new orchestration would subscribe to this trigger message and start a controller orchestration which would manage the overall process

6. The controller would iterate through a loop from 1 to the number of records in the batch and create a small simple command message which would be sent to the messagebox with an instruction to process a specific record from this batch

7. The controller orchestration would then sit polling the database until all of the records have been processed

8. For each of the command messages which had been sent to the messagebox an instance of an orchestration would be initiated which would process that particular record. If there were 5000 records you would get 5000 instances of the Record Processor orchestration

9. In the Record Processor orchestration the message it receives will tell the orchestration which batch and record number to process so it can get its data from the temporary database. This orchestration will then call the system which helps decide if the record is sent to system A or system B

10. If the record is sent to system A then the web service is called. The orchestration will then update the temporary database to indicate where the record is intended to be sent to.

11. When all records have been processed the Controller orchestration will detect this and stop polling the database.

12. The controller orchestration will extract all of the records to be sent to system B and then pass this to the messagebox so it can be mapped and delivered to system B via a port subscription.

13. The controller orchestration will query the temporary database to get the values to create the email which advises the business users how the batch was split and send it via the SMTP adapter

14. Finally the controller orchestration will clean out the temporary storage of the batch and update the sequencing ready for the next batch

Some additional points to note here are:

· If the receive pipeline gets an error then the batch would stop saving to the temporary storage. If the instance was resumed then the import would begin again from scratch

· I considered using a normal BizTalk debatching approach but encountered problems once the batch size got above 1000 records (can’t remember the error message now). Based on this I chose to implement a custom debatching solution

The Benefits

As result of the above refactoring the benefits were:

· Because the records were being processed in parallel I have been able to process a 10MB normal size file in around 15 minutes compared to the previous 6 hours

· Any errors with one record do not cause a backlog of all of the other records

· The improved performance means all of the records are loaded before the business users start their working day rather than the previous solution where they were almost finished the day before the records are all processed

· Because the messages being processed in the orchestrations are smaller than before (approximately 2kb) the fact that overall there are more persistence points (about 4 per record now) means that the total data persisted to the messagebox to maintain the state of the orchestrations has reduced from somewhere in the region of 100GB to < 40MB

Conclusion

Hopefully this post will give a real world example of how being pragmatic in a refactoring exercise can help you make significant improvements to a process with minimal cost to the project. This article may be a little unclear in how this has been implemented so if there is some interest in it the way I implemented this is fairly generic so I could potentially do a future post with an example.

Detecting BizTalk Event with BizUnit

michael stephenson — Wed, 23 Jul 2008 16:48:12 GMT

I came across an annoying one the other day, I havent had time to look into it in more detail, but here are some notes about it.

In some of the tests we do with BizUnit we sometimes check the event log to see that certain messages have occured. I came across an example I havent noticed before the other day.

In my test I do a bunch of stuff then I wait until a custom event with a specific Event Id is logged to the event log. I use the BizUnit event log check step and it finds my event fine.

Later in the test I will do the following:

Stop the application pool for a web service I will call
Allow BizTalk to call the service and then suspend the service instance because the web service is unavailable
My test will detect the event logged by BizTalk Event ID - 5754 to indicate a call has failed. I will also use the ValidationRegEx node to confirm the event message relates to my port
I will then start the application pool back up
I will then use a custom BizUnit step to resume the suspended instance
Finally I will check the process worked correctly

This was working as far as the point of detecting the BizTalk event. I could see in the event log the event I was expecting was there, but for some reason my step didnt spot it.

Because I was using the regex feature this was my first thought, but the expression was simple and even if I took it away the event wasnt spotted still.

It turns out when I wrote a little C# to check what comes back in the event log was that although the event in Event Viewer displays with the event number 5754, the event log instance from System.Diagnostics comes back with the following property values:

- Event Id: 12588666

- Instance Id: 3233814138

When I changed the event number as appropriate it worked fine. Frustrating but there you go.

Upgrading to BizTalk 2006 R2 Experience

michael stephenson — Wed, 16 Jul 2008 21:48:43 GMT

On one of the projects im associated with we have recently upgraded from BizTalk 2006 to BizTalk 2006 R2 for our next release.

In doing this we needed to upgrade our development environment and a number of testing environments. Fortunately the testing envronments are not managed by our team so we didnt need to have too much involvement in this other than providing guidance and support.

In the development environment we needed to upgrade a number of developer machines and also some build servers. We also needed to support some cycles of testing which were still happening on the non R2 machines.

Two things which you might want to consider if you are doing the same is as follows:

1. The importance of Continuous Integration

One of the first things I did was to setup the appropriate build servers to implement continuous integration so we were building and testing our code on both BizTalk 2006 and 2006 R2 at the same time.

One of the benefits of this was that it allowed me a chance to be able to deploy code which was tested in both environments and to give us more time to rebuild the developer machines.

In theory there shouldnt really be many issues for this upgrade in terms of the code we had already implemented, but using CI would help us mitigate the risk.

2. POC

Before we had begun the upgrade I had setup a machine to POC running our code in a 2006 R2 environment. Although we expected this to be trouble free I did find two small problems which needed addressing (1 a BizTalk thing and one not)

Problem with a custom pipeline component

The first problem I came across was in a custom pipeline component we have. This component helps implement a custom debatching solution in a process where we implement the splitter pattern (i will probably post something about this in the future). This component basically accepts a big message comming in, does some stuff and then replaces the message with a small trigger message which is sent to the messagebox.

The message we send out matches a schema and we set some custom context properties and the appropriate message type property. This message is then collected by an orchestration and the process continues.

While this had worked fine previously in 2006. When I ran this in BizTalk 2006 R2 I would get an error from the orchestration when it collects the mesasge and initiates the orchestration. The problem indicates that the schema strong name is incorrect.

I have come across this before and by changing our pipeline component to add the schema strong name of the message to the context (BTS.SchemaStrongName) the whole thing then worked again.

This is obviously caused by something that has changed between versions and was a simple enough fix.

Problem with one of our tests (Serialization)

The second problem was also pretty straightforward. When running the tests for one of our applications the SoapHttpRequestResponse step from BizUnit started to get an error which it hadnt previously got. This was caused because there must be some changes in the serialization functions of the ,net framework (we now have .net 2 SP1, .net 3 SP1 and .net 3.5, where we used to only have .net 2). Basically the problem was that the developer hadnt included the namespace as defined in the WSDL. While it used to work when the step used it, it now didn't

In conculsion so far the impact on our code by this upgrade has been very small and the couple of problems we have had have been eaily solved.

The one takeaway is probably my to use CI and BizUnit as it makes it much easier to detect problems before you deliver code outside of your team for testing.