Michael Stephenson

DateTime considerations for BizTalk

Recently we had a couple of defects in the development of one of our integration solutions where we hadn't handled date times correctly in relation to day light saving. When we looked for some guidance to help us it seemed that there is a fair bit of discussion around .net implications of this but little recent content about BizTalk. Probably the most useful background articles are the following:

• http://msdn.microsoft.com/en-us/library/ms973825.aspx
• http://geekswithblogs.net/dmillard/archive/2004/10/04/12117.aspx (and link to other article from here)

Although the articles talk about why things happen, I thought it would be useful to walk through some common scenarios to see how the date/time may be affected in these different situations.

Before we get into that a little about our situation and what we found (forgive the diagram).

So in our situation we have an application which publishes an event (item 1 on the diagram) which has an effective date to BizTalk. BizTalk then goes back to the application which published the event and gets some information about the entities associated with the event (item 2 on the diagram). BizTalk then sends this information to application 2 (item 3 on the diagram).

In most cases things seemed to work fine, but the enrichment data is date/time specific and we found an example when the events effective date was 26th August 2010 at midnight we were hit by date time issues. The original event which was published send a message to a web service with a date time which was 2010-08-26:00:00:00:000+01:00. In the call back to the application (again via a web service) we flowed the event effective date back to the application but in the process between the two web services the date time has suddenly changed to be 2010-08-25:23:00:00:000.

We were able to easily fix this by using the .ToLocaTime() method when constructing the message to send back to the application and because all applications are in the same time zone this fixed this for us.

What I did find when troubleshooting this is that there is only certain places where the date is converted so I thought it would be worth talking through some scenarios to elaborate on other articles which explain why this is done.

For all tests I would use the below message as the input, and I'im running the tests on a VM where the current date is 28-08-2010 and the time zone is currently British Summer Time (GMT + 1 hour).

<ns0:Root xmlns:ns0="http://Bts.DateTime.Schemas.Input">

<TestDate>2010-08-28T00:00:00.000+01:00</TestDate>

</ns0:Root>

In troubleshooting our problem I wanted to put together some scenarios so we could be clear on how a DateTime value is affected by the way it is used in different situations and hopefully the following scenarios will be useful to compliment the above articles from a BizTalk perspective.

Scenario 1: Flowing a message through a messaging scenario (Pass Through)

In this scenario I have a receive location which picks up the inbound file. There is a send port subscribing to this inbound port which puts the file back into a different folder. Both ports have pass through pipelines. The below diagram illustrates this scenario.

Test Result

In this scenario I add the message below containing the BST date of midnight on the 28th and it comes out exactly the same and the date is unchanged.

<ns0:Root xmlns:ns0="http://Bts.DateTime.Schemas.Input">

<TestDate>2010-08-28T00:00:00.000+01:00</TestDate>

</ns0:Root>

Summary

A pass through messaging scenario has no effect on the message in terms of the date time.

Scenario 2: Flowing a message through a messaging scenario (Xml Disassembler)

In this scenario I have the same as above in scenario 1 where there are two file based ports. This time the inbound port has an XmlRecieve Pipeline containing the XmlDisassembler. The below diagram illustrates this scenario.

Test Result

The output message contains the date in exactly the same format that it came in, and in the graphic below you can see that the xml disassembler promoted the date time to my property schema and it contains the date exactly as it came in containing the time zone information indicating that I am on BST.

Summary

This scenario seems to have no effect on the date time.

Scenario 3: Flowing a message through a messaging scenario (Xml Disassembler & Xml Assembler)

This scenario is an extension to scenario 2 only this time the send port has the XmlTransmit pipeline so contains the XmlAssembler component. The schema has an associated property schema so in this scenario property promotion and demotion will occur. The below diagram illustrates this scenario.

Test Result

In this case the input message results in the message coming out looking like the below.

<?xml version="1.0" encoding="utf-8"?><ns0:Root xmlns:ns0="http://Bts.DateTime.Schemas.Input">

<TestDate>2010-08-27T23:00:00.0000000Z</TestDate>

</ns0:Root>

In this case you can see the date time has been converted to the UTC format.

Summary

In this scenario there is some promotion and demotion of the date happening due to the use of the assembler and disassembler in combination with the property schema.

In this scenario you can see that the date/time is changed as it flows through BizTalk.

Scenario 4: Map on the send port

In this scenario I have a receive port with the Xml Receive Pipeline and a send port with a pass through pipeline. I have also got a map on the send port which will convert the input message into the output message. The below diagram illustrates this scenario.

Test Result

In this scenario the below result was produced.

<ns0:Root xmlns:ns0="http://Bts.DateTime.Schemas.Output"><TestDate>2010-08-28T00:00:00.000+01:00</TestDate></ns0:Root>

As you can see the date/time is still in the BST off set as was in the input file.

Summary

This scenario appears to have no impact on the date time

Scenario 5: Map on send port using Property Promotion & Demotion

In this scenario we have exactly the same situation as scenario 4 except that the send port now has the Xml Transmit Pipeline. The schemas both have an element matching to the same context property so the date will be promoted on the way in and demoted on the way out. The following diagram illustrates this scenario.

   (Update: just noticed the diagram has the wrong send pipeline, it should be XmlTransmit.  Ill update this asap).

Test Result

This test produced the below result.

<?xml version="1.0" encoding="utf-8"?><ns0:Root xmlns:ns0="http://Bts.DateTime.Schemas.Output"><TestDate>2010-08-27T23:00:00.0000000Z</TestDate></ns0:Root>

As you can see the date is in the UTC format.

Summary

In this scenario you can see the date/time is affected and really the property demotion happens after the map anyway.

Scenario 6: Xml Assembler with No Demotion

In this scenario we will use a pass through receive pipeline and an Xml Transmit pipeline. I will also use a different schema which has no association with context properties so that no promotion and demotion will take place. The following diagram illustrates this scenario.

Test Result

The following message was produced.

<?xml version="1.0" encoding="utf-8"?><ns0:Root xmlns:ns0="http://Bts.DateTime.Schemas.InputNoContext">

<TestDate>2010-08-28T00:00:00.000+01:00</TestDate>

</ns0:Root>

Summary

In this scenario the date/time is unchanged.

Scenario 7: The message flowing through an orchestration

In this scenario I will accept the input message and then this will be picked up by an orchestration. Within the orchestration I will access the date in various ways and then display the results. The below diagram shows conceptually how this works.

In this scenario my aim is to cover the following:

• Writing out the value of the date via the distinguished field
• Writing out the value of the date via the context property
• Writing out the value of the message as a whole
• Writing out the string for the date after extracting it via xpath
• Writing out the string for the date after the xpath result is parsed into a .net DateTime
• Passing the message into a helper class so the xlang message is deserialized to a class and then accessing the date as a property on the class

     

     

  The below diagram just illustrates that the orchestration is very simple. It picks up the message then I use an expression shape to access the date.

     

  

  The next picture shows the code within the expression shape.

     

  

     

     

  The below picture shows the helper class and the simple way im writing out the date from the deserialized class:

     

  

     

  Test Result

  There are a couple of observations based on the results:

     

  • The distinguished field will return the date in UTC format
  • The context property will return the date in UTC format when accessed in the expression shape even though its shown in the message context with the +1 hour off set
  • The message itself remains unchanged still containing the full date with the off set
  • Extracting the date with xpath and creating your own DateTime object will create the DateTime in the localised date format.
  • When accessing deserialized class the date is displayed in the localised format

     

     

  The below picture shows the message in the orchestrations context properties are shown below.

     

  

     

  The below picture shows the trace captured with DebugView showing the output from the different things tried in this scenario.

     

  

     

     

  Summary

  Within an orchestration there are a number of scenarios in which the date/time will be changed, but you can also control this through other techniques. This is the scenario you have to be most careful with depending on the approach you take.

     

  Conclusion

  Hopefully this article provides some useful guidance on how and when date/times may be appear to change when you implement various integration scenarios with BizTalk. A few general rules that seem to apply are:

     

  • If you allow BizTalk to assign an xsd element with a date time through property demotion then you can expect the date to be set in the UTC format.
  • If you access a date time in an orchestration via a distinguished field then you will get a .net DateTime in UTC format
  • If you access a date time context property in an orchestration then you will get a .net DateTime in UTC format
  • If you create a .net DateTime yourself based on extracting the value from a message you will get the DateTime based on the local time in the source application.

       

  Going back to the original problem our issue was caused because when we received the event message coming in from the source application we were extracting the effective date from the event message in an orchestration via a distinguished field and the creating our enrichment request message in a .net class which was then sent to the source application. In this process the localised date which came in from the source application had now converted to the UTC format when it was send back to the application which resulted in our enrichment request returning the wrong data for that customers contract. We were able to localise the date again before the enrichment to solve that problem.

     

  The source for the samples in this article is available at the following location:

  http://www.box.net/shared/91v27ugoya

posted @ Sunday, August 29, 2010 3:46 PM | Feedback (0) | Filed Under [ BizTalk ]

Do people do BizTalk Versioning?

Just wondering how many people actually do dynamic versioning of their BizTalk projects with Cruise Control or Team Build.

The reason I ask is you dont see that many things on the forums or blogs about it.  I find that it can be quite challenging at times so Im not sure if we are just doing it the hard way or its something most people dont do?

posted @ Saturday, August 14, 2010 9:08 PM | Feedback (1) | Filed Under [ BizTalk ]

Another little versioning Gotcha

Ive been having some problems with versioning for BizTalk recently and in particular it was centred around one map.

Initially the problems were related to the configuration manager but then when this was resolved there was another little gotcha which is probably worth bearing in mind.

So on the development machine everything versions as 999.999.999.999 and this is usually pretty pain free.  On the build server though a version number is applied from cruise control and replaces anything with 999.999.999.999 with the CCNet version number.  Any other version number is not touched incase we are using multiple versions of one of our assemblies.

In the map I was having problems with it always seemed to pick up either an old version or version 0.0.0.0 which was not something that should have existed.

I eventually tracked the problem down the the xml under the map where you find the following snippet referencing your assembly.

<Script Language="ExternalAssembly" Assembly="Acme.Utilities, Version=2.5.0.73, Culture=neutral, PublicKeyToken=056df4562600885d" Class="Acme.Utilities.Mapping.MemberStatusMapping" Function="Map" AssemblyPath="..\..Acme.Utilities\obj\Release\Acme.Utilities.dll" />

What is happening is that because its a project reference between the map project and the C# project its popping in the path to the utilities assembly, however when I first created the map I must have been in Release configuration where as for the other maps I must have been in Debug configuration.  In the development machine because its always the same version number this problem never shows up, but on the build server the version number changes with each build which makes this visible.

In the server build we do a Debug build and then a Release build.  In the above case when we do the Debug build there is nothing in the the release folder so it was then using the 0.0.0.0 version number.

Unfortunately this issue doesnt display itself as a compilation error which is a big pain because you get to the BizUnit tests and find that you get strange failures which are a real problem to troubleshoot and its difficult to track this back to the source of the problem.

Anyway another one which might save someone a few hours

posted @ Saturday, August 14, 2010 9:06 PM | Feedback (0) | Filed Under [ BizTalk ]

Task to detect Visual Studio Solution Configuration Issues

You may have read on my recent posts about how the Visual Studio Configuration Manager again caused me problems for my BizTalk 2006 R2 project where the release/debug/deployment/development configuration settings seem to get out of sync and it ends up causing you various problems.

I got sick of this happening so I've created an MsBuild task which I can use in my build script to inspect the solution file and highlight any issues with the configuration. Basically the task will read through the file until it finds the settings section for this and when it compares the projects it will cause the build to fail if you get any projects set as Development & Release or Deployment & Debug.

Usually the configuration you want is Development & Debug or Deployment & Release.

It's a pretty straightforward task and the code is available on the below link so rather than walking through it, if it's of interest to anyone please just refer to the unit tests in the solution to see how it works.

If you build fails you will get a list of the line numbers which aren't right and you can correct them before you end up with weird errors later in your tests which take ages to troubleshoot like in one of my previous posts.

Hope this is useful to someone

http://www.box.net/shared/ben4vap98p

posted @ Saturday, August 14, 2010 5:44 PM | Feedback (0) | Filed Under [ BizTalk ]

Unable to create the transform when your doing versioning on your build server

I had a very painful experience this week on a project.  We were implementing some new features in an existing BizTalk application and the first builds on the build server were good then suddenly a number of the tests started failing with the below error in the event log.

Basically this error is saying that the assembly the map is using to reference some functions called by the scripting functoid can not be found.

There are many forum posts along the similar lines but our situation was a little different because the build would run fine when ran locally or on the build server outside of cruise control.  It was only when cruise control runs the build and does some additional steps to apply versioning to the code that we saw this problem.

After lot of investigation I found the cause of the problem when I disassembled the map assembly using Reflector.  With reflector you can see the XSLTAgruments property which contains a string which outlines the expansion objects which will be used by the transform when executing.  This indicates the assemblies which will be called.

In this string I could see the wrong version number had been applied during the compile of the project it was just a question of working out why.

Again after a lot more digging I found that in our case it was my old friend the Visual Studio Configuration Manager which obviously decided that it hadnt been getting enough attention recently so decided to change the Debug/Release/DEployment/Development settings within the solution.  Fortunately if you have BizTalk 2009 or are planning BizTalk 2010 you shouldnt get this problem but its a big pain for 2006R2.  So what happened was that during the compilation the the BizTalk project is somehow picking up a reference to the default versioned utilities component rather than the version number supplied by cruise control.

I managed to fix this by going through all of the settings in the configuration manager and lining up the Debug --> Development and Release --> Deployment settings then everything just worked.

Im not sure how the settings get changed as it wasnt done intentionally by us and the solution file is source controlled but we had added some projects to the solution so it could have gotten in that way.

Anyway on the of chance it saves someone half a day figuring this out.

posted @ Wednesday, August 04, 2010 12:13 PM | Feedback (1) | Filed Under [ BizTalk ]

MsBuild Task for Scripting Custom Mapping Tables

Introduction

On many BizTalk projects you have to deal with managing mapping of reference data. A number of times in the past I've spoken about the features in BizTalk for mapping reference data however they do have some limitations which mean that it is very common for a project to implement their own tables in a custom database to hold mapping reference data.

While this is entirely valid you often end up with the pain of how do you manage the reference data mappings. One of the simplest ways is to just have a script with all of the statements in to create the data. This has the benefit that it can be source controlled and versioned etc, however when you get a large amount of mappings it can become a real pain, especially if you have many columns involved in the mappings.

On my current project we had exactly that problem and I decided that I would trade off the script being in source control to have a master copy of the data held in a controlled database and then during by build process I would extract the data to produce a nice tidy script which could be used to create the mappings during the deployment. My rough requirements that I wanted to implement were as follows:

• Define the tables I want to extract to a script
• Allow the master data tables to have additional columns to allow notes and stuff to be associated with a record so the extract would define which columns to extract
• The extract will plug into an MsBuild process
• I can flow a version number from the build to be written to the script
• I will clean the tables before populating the reference data

     

All fairly straightforward stuff but I thought it might be useful to others doing BizTalk or .net projects.

The MsBuild Task

In the build task I use a class generated from an xml schema shown below.

This schema defines the input definition of which tables to extract and what columns. The below pic shows an example input definition.

Some information on the attributes:

Once the task is given its parameters and definition it will iterate through the tables and produce the script to refresh the data in your database when you do a deployment activity. We produce the script during the build process and then it gets deployed with our versioned build output.

The below sample from the tests in the sample shows how to use the task, and you will need to remember to do the normal MsBuild task activities such as referencing the task etc.

The Produced Script

As you can see below the script produced by the build task has taken the data for the tables from the definition file and pulled it all together into a script which you can easily deploy.

As you can see a fairly easy task to make life a little easier for us and hopefully it will make things a little less hassle for other teams too.

Alternatives

There are tools which do database diff's such as SQL Compare and Visual Studio Database Edition, but you may not have them and in the case here I'm not actually doing a diff as the tables would not necessarily match in terms of schema because I may want additional columns to manage the data.

Source Code

The source code is available from the following link:

http://www.box.net/shared/xdsc2lgmzv

Note: Some places block box so if you have problems getting the code drop me an email

posted @ Thursday, July 22, 2010 7:18 PM | Feedback (0) | Filed Under [ BizTalk ]

BizTalk GAC Cleaning Utility

I have recently been having discussions with some of the guys from our Ops Team about the GAC and BizTalk. One of the things they had observed was that when we upgraded some of our BizTalk applications using the technique where you are able to drain the system and fully remove then replace the application the uninstallation and reinstallation process left the assemblies from the previous version in the GAC.

I explained the relationship between the management database in BizTalk and the storage of assemblies in the GAC and how assemblies in the GAC are considered shared so it is not safe in most cases to just uninstall them without some consideration over how they may be used elsewhere on the box. This is also the default behaviour of msi's which are generated by the BizTalk management console.

In our case we could be fairly sure that assemblies from old versions of our applications which are still in the GAC would not be used elsewhere I started to think about how we could clean this up in a low risk way.

I also didn't really want to get into the position of adding pre and post installation scripts to the msi at this stage as they were working well and I wanted to minimize changes.

After thinking about this for a while I've come up with the following tool:

BizTalk GAC Cleaner

The tool does the following:

1. Gets a list of all of the assemblies in BizTalk's management database

This uses the registry to get the management database for the database the server you run the tool on is connected to

2. Gets a list of all of the assemblies in the GAC

3. Compares the lists to find assemblies which can be removed from the GAC

The comparison looks for assemblies which are in both the GAC and Management database based on the assembly name and public key token. It then looks for versions of these assemblies which are in the GAC but not in the management database and then considers these assemblies to be ones which can be removed.

4. Produces a script which you can review and then run on your server to do the uninstallation   

Although I'm not a huge fan of doing this as it's not really an issue, I'm sure the odd person will have to this at some point so I've popped the source code on the below link if anyone wants it.

http://code.msdn.microsoft.com/btsgaccleaner

The sample includes versions of this for BizTalk 2006 R2. It should also work on BizTalk 2009. I used it on 2010 Beta but had to recompile it against .net 4.0 and I updated the ExplorerOM reference.

Disclaimer

Make sure you review and test your script before you run it anywhere !!!!!

And don't blame me if your referencing your assemblies from other places on the same box as I said above review the script and test it :-)

Thanks

Thanks to the below posts for the code samples for interacting with the GAC

http://blogs.msdn.com/b/junfeng/archive/2004/09/14/229650.aspx

http://blogs.msdn.com/b/junfeng/archive/2004/09/14/229653.aspx

posted @ Friday, June 18, 2010 6:40 PM | Feedback (0) | Filed Under [ BizTalk ]

BizTalk Testing Series - The xpath Function

Background

While the xpath function in a BizTalk orchestration is a very powerful feature I have often come across the situation where someone has hard coded an xpath expression in an orchestration.

If you have read some of my previous posts about testing I've tried to get across the general theme like test-driven or test-assisted development approaches where the underlying principle is that your building up your solution of small well tested units that are put together and the resulting solution is usually quite robust. You will be finding more bugs within your unit tests and fewer outside of your team.

The thing I don't like about the xpath functions usual usage is when you come across an orchestration which has something like the below snippet in an expression or assign shape:

string result = xpath(myMessage,"string(//Order/OrderItem/ProductName)");

My main issue with this is that the xpath statement is hard coded in the orchestration and you don't really know it works until you are running the orchestration. Some of the problems I think you end up with are:

• You waste time with lengthy debugging of the orchestration when your statement isn't working
• You might not know the function isn't working quite as expected because the testable unit around it is big
• You are much more open to regression issues if your schema changes

     

Approach to Testing

The technique I usually follow is to hold the xpath statement as a constant in a helper class or to format a constant with a helper function to get the actual xpath statement. It is then used by the orchestration like follows.

string result = xpath(myMessage, MyHelperClass.ProductNameXPathStatement);

This means that because the xpath statement is available outside of the orchestration it now becomes testable in its own right. This means:

• I can test it in its own right
• I'm less likely to waste time tracking down problems caused by an error in the statement
• I can reduce the risk or regression issuess

I'm now able to implement some testing around my xpath statements which usually are something like the following:   

1. The test will use a sample xml file
2. The sample will be validated against the schema
3. The test will execute the xpath statement and then check the results are as expected

      

Walk-through

BizTalk uses the XPathNavigator internally behind the xpath function to implement the queries you will usually use using the navigators select or evaluate functions.

In the sample (link at bottom) I have a small solution which contains a schema from which I have generated a sample instance. I will then use this instance as the basis for my tests.

In the below diagram you can see the helper class which I've encapsulated my xpath expressions in, and some helper functions which will format the expression in the case of a repeating node which would want to inject an index into the xpath query.

I have then created a test class which has some functions to execute some queries against my sample xml file. An example of this is below.

In the test class I have a couple of helper functions which will execute the xpath expressions in a similar way to BizTalk. You could have a proper helper class to do this if you wanted.

You can see now in the BizTalk expression editor I can use these functions alongside the xpath function.

Conclusion

I hope you can see with very little effort you can make your life much easier by testing xpath statements outside of an orchestration rather than using them directly hard coded into the orchestration.

This can also save you lots of pain longer term because your build should break if your schema changes unexpectedly causing these xpath tests to fail where as your tests around the orchestration will be more difficult to troubleshoot and workout the cause of the problem.

Sample Link

The sample is available from the following link:

http://code.msdn.microsoft.com/testbtsxpathfunction

Other Tools

On the subject of using the xpath function, if you don't already use it the below tool is very useful for creating your xpath statements (thanks BizBert)

http://www.bizbert.com/bizbert/2007/11/30/XPath+The+Hidden+Language+Of+BizTalk.aspx

posted @ Sunday, June 13, 2010 4:41 PM | Feedback (0) | Filed Under [ BizTalk ]

Connected Systems User Group Meeting tomorrow

Just wanted to pass out a reminder about tomorrow nights user group meeting with Udi Dahan

Registration link:http://sbugmay2010.eventbrite.com/

User group link: http://ukconnectedsystemsusergroup.org

posted @ Tuesday, May 25, 2010 5:20 PM | Feedback (0) | Filed Under [ BizTalk ]

Multi-Part Map Troubleshooting

Scenario

I came across a nice little one with multi-part maps the other day. I had an orchestration where I needed to combine 4 input messages into one output message like in the below table:

I thought my orchestration was working fine but for some reason when I was trying to send my message it had no content under the root node like below

<ns0:ImportMemberChange xmlns:ns0="http://---------------/"></ns0:ImportMemberChange>

My map is displayed in the below picture.

I knew that the member search message may not have any elements under it but its root element would always exist. The rest of the messages were expected to be fully populated.

I tried a number of different things and testing my map outside of the orchestration it always worked fine.

The Eureka Moment

The eureka moment came when I was looking at the xslt produced by the map. Even though I'd tried swapping the order of the messages in the input of the map you can see in the below picture that the first part of the processing of the message (with the red circle around it) is doing a for-each over the GetCompanyDetailsResult element within the GetCompanyDetailsResponse message.

This is because the processing is driven by the output message format and the first element to output is the OrganisationID which comes from the GetCompanyDetailsResponse message.

At this point I could focus my attention on this message as the xslt shows that if this xpath statement doesn’t return the an element from the GetCompanyDetailsResponse message then the whole body of the output message will not be produced and the output from the map would look like the message I was getting.

<ns0:ImportMemberChange xmlns:ns0="http://---------------/"></ns0:ImportMemberChange>

I was quickly able to prove this in my map test which proved this was a likely candidate for the problem.

I revisited the orchestration focusing on the creation of the GetCompanyDetailsResponse message and there was actually a bug in the orchestration which resulted in the message being incorrectly created, once this was fixed everything worked as expected.

Conclusion

Originally I thought it was a problem with the map itself, and looking online there wasn’t really much in the way of content around troubleshooting for multi-part map problems so I thought I'd write this up.

I guess technically it isn't a multi-part map problem, but I spend a good couple of hours the other day thinking it was.

posted @ Saturday, May 22, 2010 3:53 AM | Feedback (0) |

Versioning and Continuous Integration with project settings files

I came across something which was a bit of a pain in the bottom the other week.

Our scenario was that we had implemented a helper style assembly which had some custom configuration implemented through the project settings. I'm sure most of you are familiar with this where you end up with a settings file which is viewable through the C# project file and you can configure some basic settings.

The settings are embedded in the assembly during compilation to be part of a DefaultValue attribute. You have the ability to override the settings by adding information to your app.config and if the app.config doesn’t override the settings then the embedded default is used.

All normal C# stuff so far…

Where our pain started was when we implement Continuous Integration and we wanted to version all of this from our build. What I was finding was that the assembly was versioned fine but the embedded default value was maintaining the non CI build version number.

I ended up getting this to work by using a build task to change the version numbers in the following files:

• App.config
• Settings.settings
• Settings.designer.cs

I think I probably could have got away with just the settings.designer.cs, but wanted to keep them all consistent incase we had to look at the code on the build server for some reason.

I think the reason this was painful was because the settings.designer.cs is only updated through Visual Studio and it writes out the code to this file including the DefaultValue attribute when the project is saved rather than as part of the compilation process. The compile just compiles the already existing C# file.

As I said we got it working, and it was a bit of a pain. If anyone has a better solution for this I'd love to hear it

posted @ Saturday, May 22, 2010 3:49 AM | Feedback (0) |

Testing Reference Data Mappings

Background

Mapping reference data is one of the common scenarios in BizTalk development and its usually a bit of a pain when you need to manage a lot of reference data whether it be through the BizTalk Cross Referencing features or some kind of custom solution.

I have seen many cases where only a couple of the mapping conditions are ever tested.

Approach

As usual I like to see these things tested in isolation before you start using them in your BizTalk maps so you know your mapping functions are working as expected.

This approach can be used for almost all of your reference data type mapping functions where you can take advantage of MSTests data driven tests to test lots of conditions without having to write millions of tests.

Walk Through

Rather than go into the details of this here, I'm going to call out to one of my colleagues who wrote a nice little walk through about using data driven tests a while back.

Check out Callum's blog: http://callumhibbert.blogspot.com/2009/07/data-driven-tests-with-mstest.html

posted @ Saturday, May 22, 2010 3:46 AM | Feedback (0) |

Using NServiceBus behind a custom web service

In this post I'd like to talk about an architecture scenario we had recently and how we were able to utilise NServiceBus to help us address this problem.

Scenario

Cognos is a reporting system used by one of my clients. A while back we developed a web service façade to allow line of business applications to be able to access reports from Cognos to support their various functions.

The service was intended to provide access to reports which were quick running reports or pre-generated reports which could be accessed real-time on demand. One of the key aims of the web service was to provide a simple generic interface to allow applications to get any report without needing to worry about the complex .net SDK for Cognos.

The web service also supported multi-hop kerberos delegation so that report data could be accesses under the context of the end user.

This service was working well for a period of time.

The Problem

The problem we encountered was that reports were now also required to be available to batch processes. The original design was optimised for low latency so users would enjoy a positive experience, however when the batch processes started to request 250+ concurrent reports over an extended period of time you can begin to imagine the sorts of problems that come into play. The key problems this new scenario caused are:

1. Users may be affected and the latency of on demand reports was significantly slower
2. The Cognos infrastructure was not scaled sufficiently to be able to cope with these long peaks of load

From a cost perspective it just isn't feasible to scale the Cognos infrastructure to be able to handle the load when it is only for a couple of hour window each night. We really needed to introduce a second pattern for accessing this service which would support high through-put scenarios.

We also had little control over the batch process in terms of being able to throttle its load. We could however make some changes to the way it accessed the reports.

The Approach

My idea was to introduce a throttling mechanism between the Web Service Façade and Cognos. This would allow the batch processes to push reports requests hard at the web service which we were confident the web service can handle. The web service would then queue these requests and process them behind the scenes and make a call back to the batch application to provide the report once it had been accessed.

In terms of technology we had some limitations because we were not able to use WCF or IIS7 where the MSMQ-Activated WCF services could have helped, but we did have MSMQ as an option and I thought NServiceBus could do just the job to help us here.

The flow of how this would work was as follows:

1. The batch applications would send a request for a report to the web service
2. The web service uses NServiceBus to send the message to a Queue
3. The NServiceBus Generic Host is running as a windows service with a message handler which subscribes to these messages
4. The message handler gets the message, accesses the report from Cognos
5. The message handler calls back to the original batch application, this is decoupled because the calling application provides a call back url
6. The report gets into the batch application and is processed as normal

This approach looks something like the below diagram:

The key points are an application wanting to take advantage of the batch driven reports needs to do the following:

• Implement our call back contract
• Make a call to the service providing a call back url

• Provide a correlation ID so it knows how to tie each response back to its request

What does NServiceBus offer in this solution

So this scenario is not the typical messaging service bus type of solution people implement with NServiceBus, but it did offer the following:

• Simplified interaction with MSMQ
• Offered the ability to configure the number of processes working through the queue so we could find a balance between load on Cognos versus the applications end to end processing time
• NServiceBus offers retries and a way to manage failed messages
• NServiceBus offers a high availability setup

The simple thing is that NServiceBus gave us the platform to build the solution on. We just implemented a message handler which functionally processed a message and we could rely on NServiceBus to do all of the hard work around managing the queues and all of the lower level things that would have took ages to write to any kind of robust level.

Conclusion

With this approach we were able to deal with a fairly significant performance issue with out too much rework. Hopefully this write up gives people some insight into ideas on how to leverage the excellent NServiceBus framework to help solve integration and high through-put scenarios.

posted @ Thursday, May 20, 2010 6:09 PM | Feedback (2) |

Humerous Word 2010 "feature"?

Im just sitting on the train to work and had a funny experience with word 2010 that I thought id share.

Im writing a document and all of a sudden like usually happens the train gets a little bit bumpy.  Word decides it doesnt like this (maybe it prefers to fly?).  Anyway to show its dissatisfaction with the journey it starts adding new rows to my table in the document all by itself.

5 pages of rows later I still cant workout how to stop itso have to kill word.

Thank you autosave

posted @ Sunday, May 09, 2010 4:27 PM | Feedback (2) |

NServiceBus Generic Host and mqsvc.exe high CPU

UPDATE

Just to provide an update on this Udi Dahan advised me that:

This has now been fixed on the 2.0 branch and on the trunk and a fatal-level log entry is now written and the process then exits after a couple of seconds.

We have been doing some work with NServiceBus recently and observed some unusual behaviour which was caused by our mistake and seemed worthy of a small post.

The Scenario

In our solution we were doing some standard NServiceBus stuff by pushing a message to a queue using NServiceBus.  We had a direct send/receive scenario rather than a publish/subscribe one.

The background process which was meant to collect the message and then process it was a normal NServiceBus message handler.  We would run the NServiceBus.Host.exe which would find the handler and then do the usual NServiceBus magic.

The Problem

In this solution we were creating some automated tests around this module of the integration process to ensure that it would work well.  We had two tests.

Test 1

This test would start NServiceBus.Host.exe using the Process object, then seed a message to the queue via our web service façade sitting above the queue which wrapped NServiceBus.  The background process would then process the message and the test would check the message had been processed fine.

If all was well then the NServiceBus.Host.exe process was stopped.

Test 2

In test 2 we would do a very similar thing except that instead of starting the process the test would install NServiceBus.Host.exe as a windows service and then start the service before the test and once the test was executed it would stop the test.

The Results of the Tests

Test 1 worked really well, however in test 2 we found that it didn’t really work at all, instead of doing the background process we were finding that between mqsvc.exe and NServiceBus.Host.exe the CPU on the machine was maxed and nothing was really happening.

The Solution

After trying a few things we found it was the permissions on the queue were not set correctly.  Once this was resolved it all worked fine and CPU was not excessive and ran just like the console application.

I think the couple of take aways from this are:

• Make sure you set the windows service for NserviceBus Generic Host to the right credentials

When you install the generic host as a windows service then by default it will use the default windows credentials.  For any production like scenario you should be using a domain account to run the process as via the windows service.

• Make sure you have the queue set with the right permissions

For the credentials you have used to configure the generic host as a windows service you should ensure that this user has the appropriate permissions for any queues it will interact with.

• Make sure you turn on the right logging configuration in NServiceBus

When this wasnt working correctly we didnt know there was an issue, we were just experiencing the high CPU condition.  I am a little surprised that there wasnt something logged and that the process didnt crash.  I guess this could be by design bearing in mind that the process could be monitoring many queues.  In this point Im just saying that originally we didnt have all of the log4net logging which is available from NServiceBus turned on.  Its probably a good idea to have this turned on and configured until you are happy your solution is working fine.

Thanks to Ahmed Hashmi on my team who got this working in the end.

posted @ Friday, May 07, 2010 5:01 PM | Feedback (4) |

UK Connected Systems User Group - Udi Dahan Event Topic change

Hi

Just wanted to get the word out about a change to the may user group event.  Udi Dahan will present a new topic which he has not presented in the UK before.  Details below.

To register for this event please refer to: http://ukconnectedsystemsusergroup.org/UpcomingEvents.aspx

Title: High Availability - A Contrarian View

Abstract: Many developers are aware of the importance of high availability, critically analyzing any single points of failure in the infrastructure. Those same developers rarely give a second thought to the periods of time when a system is being upgraded. Even if all the servers are running, most systems cannot function in-between versions. Yet with the increased pace of business, users are demanding ever more frequent releases. The poor maintenance programmers and system administrators are left holding the bag long after the architecture that sealed their fate was formulated. Join Udi for some different perspectives on high availability - architecture and methodology for the real world.

posted @ Monday, April 19, 2010 5:52 PM | Feedback (0) |

User Group Meeting Summary - April 2010

Thanks to everyone who could make it to what turned out to be an excellent SBUG event.  First some thanks to: 

1. Speakers: Anthony Cooper and Elton Stoneman
2. Host: The various people at Hitachi who helped to organise and arrange the venue.

    

   Session 1 - Getting up and running with Windows Mobile and the Windows Azure Service Bus

   In this session Anthony discussed some considerations for using Windows Mobile and the Windows Azure Service Bus from a real-world project which Hitachi have been working on with EasyJet.  Anthony also walked through a simplified demo of the concepts which applied on the project.

    

   In addition to the slides and demo it was also very interesting to discuss with the guys involved on this project to hear about their real experiences developing with the Azure Service Bus and some of the limitations they have had to work around in Windows Mobiles ability to interact with the service bus.

    

   On the back of this session we will look to do some further activities around this topic and the guys offered to share their wish list of features for both Windows Mobile and Windows Azure which we will look to share for user group discussion.

    

   Another interesting point was the cost aspects of using the ISB which were very low.

    

   Session 2 - The Enterprise Cache

   In the second session Elton used a few slides which are based around one of his customer scenario's where they are looking into the concept of an Enterprise Cache within the organisation.  Elton discusses this concept and also a codeplex project he is putting together which allows you to take advantage of a cache with various providers such as Memcached, AppFabric Caching and Ncache.

    

   Following the presentation it was interesting to hear peoples thoughts on various aspects such as the enterprise cache versus an out of process application cache.  Also there was interesting discussion around how people would like to search the cache in the future.

    

   We will again look to put together some follow-up activity on this

    

   Meeting Summary

   Following the meeting all slide decks are saved in the skydrive location where we keep content from all meetings:

   http://cid-40015ea59a1307c8.skydrive.live.com/browse.aspx/.Public/SBUG/SBUG%20Meetings/2010%20April

    

   Remember that the details of all previous events are on the following page.

   http://uksoabpm.org/Events.aspx

    

   Competition

   We had three copies of the Windows Identity Foundation Patterns and Practices book that were raffles on the night, it would be great to hear any feedback on the book from those who won it.

    

   Recording

   The user group meeting was recorded and we will look to make this available online sometime soon.

    

   UG Business

   The following things were discussed as general UG topics:

    

3. We will change the name of the user group to the UK Connected Systems User Group to we are more inline with other user groups who cover similar topics and we believe this will help us to attract more members.  The content or focus of the user group is not expected to change.

    

4. The next meeting is 26th May and can be registered at the following link: http://sbugmay2010.eventbrite.com/

    

posted @ Sunday, April 18, 2010 3:35 PM | Feedback (0) |

SBUG --> UK Connected Systems User Group

Following a recent user group meeting we have decided that the UK SOA/BPM User Group will be renamed to the UK Connected Systems User Group.  The reasons for this are as follows:

1. Other user groups who cover the same topics as us are all called something similar

2. We feel the name change will help to increase user group membership

The focus and topics of the user group will remain the same.

posted @ Sunday, April 18, 2010 3:29 PM | Feedback (0) |

I want to hit Apex SQL with a big stick

<Whinge>

Thought id just have a little whinge about this product which caused me a load of grief the other day.....

So the background was that my development machine had a completely full hard disk which I needed to sort out.  Upon investigation I found the issue was that the msdb database had managed to get very large.

This was caused because a long time ago (and I cant even remember why) I tried out Apex SQL.  After a few days I decided to uninstall it and thought nothing more of it.  What I didnt realise was that uninstalling it doesnt actually uninstall it (and it doesnt inform you about this), but there was still some assemblies left on my machine.  Everytime SQL Server was running it was starting the Apex SQL Connection monitor which was then running in the background and regularly recording information in the msdb database.  Over time it had recorded enough to fill the disk.

The below article advises how to sort this out by removing this fully so if your having a problem then try this out:http://knowledgebase.apexsql.com/2007/08/how-to-uninstall-apexsqlconnectionmonit_09.htm

Once this was sorted out its interesting to read the above article because I just dont think the approach used by the vendor of this software is a very good one.  So for the Apex team just wanted to pass on a thought:

If I want to uninstall your product you should tell me if stuff is left on the machine especially if a process will be running which is going to fill my machine with useless data,

</Whinge>

posted @ Wednesday, April 14, 2010 7:08 PM | Feedback (1) |

AppFabric Caching Feedback

We are running a survey to collect feedback around scenarios where people were experimenting with velocity on windows 2003 in the CTP but are now in a position where the beta requires windows 2008.

We would like to understand how important this scenario is precieved to be.

If you are in the Connected Systems Community and would like to provide feedback please complete the following survey

http://www.surveymonkey.com/s/N3VKZWN
 

posted @ Tuesday, March 16, 2010 7:03 PM | Feedback (0) |

BizTalk & NService Bus Whitepaper

My whitepaper discussing BizTalk and NService Bus is now available on MSDN

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=b57b7625-7316-4f56-b88e-1fb685efae5b

Thanks to Steve Lemkau for his contribution and to Udi Dahan and the NServiceBus User Forums for help with a few questions I had.

posted @ Monday, March 15, 2010 9:10 PM | Feedback (2) |

SBUG April 2010

We are just putting together the details for the SBUG meeting in April, the registration page is already open on the following link

http://sbugapril2010.eventbrite.com/

posted @ Saturday, February 27, 2010 5:42 PM | Feedback (1) |

SBUG May 2010

We have an SBUG meeting in may planned where Udi Dahan (nServiceBus) will join us.

Check out the following for details:

http://sbugmay2010.eventbrite.com/

posted @ Saturday, February 27, 2010 5:41 PM | Feedback (0) |

HP OVO Whitepaper

A couple of people have asked me recently where the whitepaper is... its here:

http://go.microsoft.com/fwlink/?LinkId=183500

posted @ Wednesday, February 17, 2010 1:15 AM | Feedback (1) |

BizTalk & HP Openview Whitepaper Complete

Ive eventually managed to get the "Monitoring BizTalk with HP Openview" whitepaper complete and submitted to Microsoft to go through the publishing process.  Not sure how long this takes?

Anyway it should be around soon

Big thanks to Saravana Kumar (BizTalk 24/7) and Dave Gerrish (a great HP Openview Consultant) for their input, and to Elton Stoneman for reviewing it

Cheers

Mike

posted @ Sunday, February 07, 2010 1:17 PM | Feedback (2) |