BizTalk 24*7

Saravana has been doing lots of work on this site.  Check it out its really cool!

http://www.biztalk247.com

posted @ Monday, May 25, 2009 10:20 AM | Feedback (0) | Filed Under [ BizTalk ]

SBUG Meeting next Tuesday

Just a quick reminder for the SBUG meeting next Tuesday.  We will be joined by:

-          Tony Meleg from the Dublin Team doing an overview of Dublin following on from the TechEd Dublin sessions

-          John Plummer from Microsoft UK hosting a Q&A session

Remember this is an in person meeting at Cardinal Place, but for those who cant make it in person we will also have live meeting available so you can still log in and join  us.

The full event details are on the following page:

http://sbug20090526.eventbrite.com/

posted @ Thursday, May 21, 2009 11:08 PM | Feedback (0) |

BizTalk Testing Guidance - Stub Of Dynamics AX

Following on from my previous article, one of my old colleagues has written up an article about how a project he worked on was able to simulate the behaviour of Dynamics AX by using one of the approaches I described in my article about how to simulate external systems in testing.

Check out Charlies article here: http://geekswithblogs.net/charliemott/archive/2009/05/13/stub-of-dynamics-ax-for-biztalk-development-and-bizunit-testing.aspx

posted @ Wednesday, May 13, 2009 4:57 PM | Feedback (0) |

BizTalk Testing Guidance - Dealing with external systems/dependancies

Article source: http://geekswithblogs.net/michaelstephenson

When we are testing our BizTalk solution one of the biggest challenges is how we deal with the dependency on external systems. Some of the common scenarios I have come across are as follows:

1. We will be integrating with some web services but there are no test instances of these services we can use
2. We will integrate with a LOB application but there is either no test instance of the LOB system of using the LOB application will make it difficult to perform a lot of our testing
3. We are working on a larger project and the system we will be integrating with does not yet exist or is in early development and is changing a lot
4. I'm working with a B2B partner and they can not support my testing

These problems are typical on most projects and are one of the common excuses for poor testing and a resulting low quality solution. In this article I will discuss some of the approaches we take to dealing with these problems.

Probably one of the key things here is that if you are working with BizTalk or some kind of services solution you will hopefully be thinking carefully about a contract first approach. By using a contract first approach you will understand the messages you will be passing between you and external systems and at least how your solution will react to different messages it receives. With this understanding you should be able to use various techniques to simulate how an external system will work in your solution.

Obviously there is a trade off because you don't want to recreate the whole external application but if you can mimic the communication protocol and replicate some of the messaging exchanges that will be involved in some of your testing scenarios then you can make significant improvements in the testing you can do.

Use a Mocking Framework

For completeness I will mention the use of mocking frameworks such as Rhino Mocks. While these frameworks are good and offer a lot of value in the testing of some aspects of BizTalk solutions I think when it comes to BizUnit style testing of an end to end process they don't really help you.

I usually use something like Rhino Mocks to test a custom .net component, pipeline component or possibly a custom adapter. I have mentioned these in previous posts in the testing series.

I will later in this article discuss a community project called BizMock which I feel is slightly different to NMock and Rhino Mocks.

Creating a Stub

One of the simplest ways you can replicate the external system is by creating a simple stub of that system. One of the most common ways to do this if it is a web services enabled application is to create a Visual studio web services project in your solution and then add some web services and web methods that match the external web service. I generate a server side stub of the web service and then add some logic to the generated method to make it meet the test cases. Often the stub will implement some logic to return different responses based on a parameter in the inbound request message or do a different action.

I will provide a couple of follow-up articles to show how we normally create stubs for asmx web services and WCF services.

I have made a list of what I think are some of the main good and bad points about this approach:

I have an example of a situation where a stub approach really made our life easier. In a project we were developing a web service façade on top of the API for the Cognos Reporting system. The intention was to provide a flexible service which applications could call to execute a report in Cognos and the service would encapsulate all of the logic required to execute the report.

The Cognos API is a large assembly which allows you to do a lot of different things against Cognos. One of these things is to execute a report. The report execution is slightly complex and to get good coverage of our code was going to be slightly difficult. One of the challenges was that we would need an instance of Cognos to be able to connect to. This would be a pain as there were a lot of Cognos projects going on and getting on of their boxes with a set of suitable test reports which they could maintain wouldn't really work because as soon as there is a conflict between team priorities someone is going to be affected by this coupling.

We considered having a local instance of Cognos to support our development/build/test effort but straightaway this requires our team to have a Cognos skill set.

With a little bit of investigation we worked out that the Cognos API used to send HTTP traffic between itself and the Cognos server so we were able to capture some of this traffic and to use this to simulate the HTTP calls so we could then create a .net Web Application which used an aspx page to receive and send the appropriate messages to allow the Cognos client side API to work. The benefits of this stub were:

• We didn't need to depend on a Cognos installation for all of our development effort
• We were able to test parts of the code which we couldn't without working out how to break Cognos from our test case
• We didn't need to create any test reports
• We didn't need any Cognos skill

This was probably one of the more extreme cases of stubbing out an application. We had initially depended on a Cognos instance but we found that our Continuous Integration builds would constantly break because someone deleted our reports etc.

Using Mocking Bird

Mocking Bird is a codeplex project that was introduced to me by Santosh Benjamin. It is something I haven't used in anger but it's been on my radar for a while. Basically the project provides a pattern to allow you to implement stubs to simulate external systems. This project can simulate traditional web services and WCF services.

I think it will save you some of the effort of creating and maintaining the stubs discussed above.

Santosh has some good articles about Mocking Bird and I would recommend looking into it. Probably the one key area where it will not help is if your external application is not supported by WCF/Web services. In this scenario you can just drop back to the above stub approach and code something to simulate that system.

For more info refer to: http://www.codeplex.com/mockingbird

Using Alternative Bindings

I was providing some advice to a project once and they were doing some integration with Dynamics AX (I think). Anyway for whatever reason they could not integrate with web services and were using the BizTalk AIF Adapter for Dynamics AX. One of the dependencies of the adapter was that you needed a build and ready deployment of Dynamics AX for the adapter to work with.

On the project at the same time that the BizTalk team were developing an EAI solution, a Dynamics team were also working hard on the Dynamics part of the project. This means that this coupling would cause problems with the BizTalk team not being able to test against Dynamics and it being difficult to stub out the LOB application due to the complexity of the adapter.

One of the advantages of BizTalk is that because you have loose coupling between an orchestration and any ports it uses the BizTalk team in this scenario were able to change their approach to testing in their development environment and use different bindings to change the communication protocol to talk to a stub which was created to simulate the behaviour of Dynamics but it would use a different communication mechanism.

By looking at the message exchange patterns the BizTalk team were able to use File ports to simulate 1 way communications and WCF ports to simulate 2 way communication patterns. This means that in the development tests the team could use BizUnit to effecticely test the solution with these bindings and focus on the logic in their orchestrations and other components.

The one big risk with this approach is that you are not actually testing connectivity to the LOB application and some of the vendor specific situations may result in problems being faced later in testing. There isn't much you can do about this but you have certainly ensured you are doing an effective job at testing your own code.

I think this model would also be suitable for off-shoring situations where your off shore resource may not have access to the LOB applications to help with testing.

In summary this approach basically uses a different type of port in the bindings for different test environments.

Using BizMock

I came across BizMock a few weeks ago and it seems like a really interesting community project that could be a valuable resource for those who develop BizTalk solutions. I asked one of my team to have a look at BizMock and to try to replicate some of our existing BizUnit tests with BizMock to see what it could offer us.

Unfortunately we were getting the same error that has already been logged on the BizMock issue tracker so we haven't been able to complete a full comparison yet. BizMock is a very new Codeplex project in its alpha release so being we can live with this, but the ideas that the project offers are very interesting.

The below blog article provides a good overview of what BizMock will offer.

http://blogs.msdn.com/pierreml/archive/2009/02/13/bizmock.aspx

My opinion on BizMock is that I would use it in situations where I have a lot of complex logic in an orchestration which I need to test and external dependencies make this difficult. BizMock looks to be aimed at testing orchestrations rather than an end to end process so it has a clear differentiator to BizUnit. BizMock also offers schema and map testing capabilities but with the enhancements in BizTalk 2009 I'm not sure what value they would offer.

Once there is a more stable release of BizMock I would use it to compliment the tests I currently do in BizUnit. I think managing another set of bindings to configure the BizMock ports is something which I don't particularly want to do so the trade off of BizMock vs Stubs is just work saved in 1 area to be spent in another. I think the real benefit of BizMock is in being able to concentrate on those test scenarios which are difficult to perform or require lots of manipulation of a stub.

I think in the feature list in the above lined article it discusses the benefits of the tests being written in code vs in xml. Remember in the most recent version of BizUnit you can also write tests in C#, but to be honest I think I actually prefer them in XML where they are much clearer and easier to follow and have a more structured approach and you don't have to worry about some developer who is no longer around who had a coding style that no one can follow.

I'm looking forward to a resolution to the problem logged on Codeplex so we can look into BizMock in more detail after which I will update this section.

More info is available on: http://www.codeplex.com/bizmock

Summary

Hopefully in this article you got a good feeling for some of the approaches that are available to help you with some of the common yet challenging aspects of BizTalk testing. The things I've discussed here are primarily focused around the development cycle where you are trying to perform automated testing as part of a continuous integration approach.

In general my comments were that Stubs are a good way to help you test your solution and deal with the parts of your project that have dependencies to external systems. When you have very tight coupling to an external system you are helped by BizTalk in that you can use a different type of port to reduce this coupling to do some testing with a trade off that you take on some risk of adapter specific implications.

I think Mocking Bird and BizMocks offer some interesting new ideas on how to look at these problems.

One thing that I haven't mentioned is that in addition to this focused development testing it is also very important to get your real systems together on a regular basis and perform some integration testing. Ideally this would be automated but it can be effective even if its just getting a couple of developers from the different teams together on a regular basis. You will always get some problems this is why integration is usually such a pain but with these approaches we are making our best effort to catch problems early and mitigate as much risk as possible.

posted @ Tuesday, May 12, 2009 12:13 PM | Feedback (1) |

BizTalk RFID Workshop Review

Earlier this week I attended the RFID workshop held at Microsoft's Reading Offices. I thought Id write up some notes and thoughts I'd made about the event.

Content/Trainer

The course was based on the course developed by the guys at Breeze (http://breeze.net/default.aspx). The material itself is excellently written and is easy to follow. It also has some nice little bits of humour which mean it is not the common/boring material you get on some courses. The material itself lets you build and enhance an RFID solution through out the course of the labs and there are about 10 labs and the course was very hands on.

The trainer for our course was Jeff Johnson from Microsoft in the UK and he delivered the course very well and was able to add lots of additional information based on experiences from work he had done.

The course itself was a 2 day workshop and was free.

Freebies

There were some excellent freebies on this course:

• A copy of the BizTalk 2009 RFID book
• A USB RFID reader/writer
• A small bag of RFID tags
• A BizTalk VPC (2006 R2) with everything on to do additional study away from the course

Attendees

The attendees for the workshop gave us an excellent mix of BizTalk people who had done little or no RFID work and RFID specialists who hadn't done much with BizTalk.

This resulted in lots of discussion and interesting points based on peoples differing experiences. It was also good to see some people from the UK SBUG user group there.

My Random Thoughts based on what I learnt on the course

I think the first thing to say is that I have wanted to look into the RFID aspects of BizTalk for quite a while, but the combination of me not having any RFID work in the pipeline and my expectation that it would be quite a steep learning curve and that I didn't have the hardware I would need to play around with it had put me off doing this for a long time. It just kept getting reprioritised.

Jeff had actually contacted me to promote the workshop to our user group members and I thought it was a good time to do some training and being free it ment I would only lose the cost of the days off work.

To my surprise RFID work with BizTalk is sooooo much easier than I expected, so much so that I had to question if it was appropriate to call it BizTalk being so easy.

I don't really want to go into too much discussion about the aspects of RFID but some things which I feel will catch people's interest to look more into this might be:

1. Traditionally RFID work had the typical complications of working with hardware vendors. You needed different API's for each vendors readers etc. BizTalk RFID abstracts this and gives you a standardised API which allows you to communicate with any supported vendor. You could really see the benefits of this and you can code directly against this API from .net applications if required

    

2. BizTalk RFID is actually a separate module to the main BizTalk installation. The course really discussed setting up RFID on "edge" servers which would be configured to work with a set of readers. You could then send your events to a central hub BizTalk installation to interact with LOB applications if required

    

3. Following on from the above comment it really looks like you would use RFID with the branch edition of BizTalk on your edge server. This significantly reduces the cost of setting up solutions and also if I remember correctly you don't need a hub BizTalk enterprise edition server if you don't need one. This allows customers who may use other vendors for their integration technologies to still take advantage of BizTalk RFID as a low cost solution to work with their current products, it also offers SME companies a cheap option and as they grow it gives them the option to bring "proper" BizTalk into the picture later.

    

4. BizTalk RFID doesn't require the traditional BizTalk skill set. There is some configuration requirements to configure your devices and RFID processes, but when it comes to custom processes the components are written in .net code in a similar way to BizTalk pipelines and pipeline components (but much simpler!!)

    

5. The RFID module gives you an execution environment for processing your RFID events and you can have multiple processes which will pick up the same events depending on how you configure your bindings from devices to processes.

    

6. The configuration/administration side of things is quite BizTalk like but with out any of the SQL Server requirements

    

7. There are some testing things that come with RFID which allow you to test your processes without having to have an RFID reader

    

I'm sure that there is a whole lot of other stuff that I should say but just doesn't come to mind at the moment.

In terms of recommending the course I would say that it is a very good course and it has really excited me about the possibilities of doing BizTalk RFID solutions.

I have invested 2 days of time and I'm coming away with a good understanding of this and I think with very little effort I can setup a pretty powerful demo to show my customers.

I believe Microsoft might be running this workshop again in the Autumn or if you are outside the UK I'm sure Breeze can advise you how to do this course.

posted @ Thursday, April 30, 2009 8:13 PM | Feedback (1) |

Poor visual studio integration

<Rant>

I came across the worst example of visual studio source control integration ever today.  I was checking in some work and added the comment about the check in in the pending checkins box.  I then pressed check in.

We use MKS source control (unfortunately) which then preceeded to display 75 confirmation boxes confirming that I was sure i wanted to check in and that I wanted to use the comment I had supplied in Visual Studio.

Note to anyone doing Visual Studio source control integration:

1. Integration into visual studio means that as a developer I dont really want to know much if anything about the source control system under the hood

2. Yes means yes.  When I confirm that I want to check in dont ask me over and over again.

</Rant>

posted @ Wednesday, April 22, 2009 10:36 PM | Feedback (0) |

How BizTalk handled some strange FTP behaviour

We had a little problem a few days ago when we were reviewing the testing of a B2B solution implemented with BizTalk.

This implementation was basically the collection of a file via FTP and then a splitter pattern which would break up a batch and cascade updates to appropriate systems.  The file which was recieved was a moderately complex file containing multiple rows containing different types of positional records.

We had implemented this as per the specification and were moving from testing this internally to test it with the business partner.  One of our limitations was that we could not do integration testing as early as we would like due to some external constraints.

Our internal tests were fine, but during integration testing we found some unexpected behaviour from the partners FTP service.  A summary of this behaviour is:

• The server would store multiple files with the exact same file name.  If the partner uploaded 2 files we could see them as seperate instances with seperate file creation dates, but they would both have the same name.
• When we executed a GET command on a file it would get the file and then mark it as collected and prevent us from downloading the file again.  To be able to get a file if transmittion had failed we need the partner to make the file available again
• We are unable to delete files from the server
• If 2 files with the same name can be seen on the remote server, if we execute a get it will actually merge the files together to give us one file locally containing all of the date from both remote files. The data positions the content of the files in date order

We experienced a little problem and became aware of this unexpected behaviour and our first thoughts were that we would need to do some custom coding to deal with this.  On closer inspection our problem seemed to be something else and BizTalk was actually dealing with this FTP behaviour in a way that worked well for us.  Our setup basically had the FTP adapter polling the remote server when files were downloaded they were streamed through a custom pipeline which contained the FFDasm pipeline component which used a schema based on the partners message specification.

The way BizTalk was dealing with this additional FTP requirements was as follows:

• It didnt matter that BizTalk couldnt delete the file from the remote server because once the file was downloaded it was no longer available anyway.
• When 2 or more files were merged together when downloaded the FFDasm component was recognising this and actually still broke the message up correctly.  If 2 files were downloaded and merged together we would actually get 2 disassembled messsages in the messagebox which had been handled correctly by FFDasm

I guess in hindsight this kind of makes sense, but it was nice to come across a situation where something like this happens and you done end up having to pull together some hack to workaround it.

In terms of the partners FTP service we didnt get confirmation on the vendor specific setup but based on some googling (sorry I mean Live searching) I believe that the it could be based on a VAX file system where you can have multiple versions of a file and the FTP service could be either HP/UX Enterprise Secure FTP or possible an offering from Sterling

posted @ Friday, April 17, 2009 12:05 AM | Feedback (2) |

BizTalk Blog Docs

Saravana Kumar has just published the BizTalk Blog Docs site on BizTalk 24/7.

This is a really cool looking new resource which aggregated some of the popular blog articles and presents them in a structured format which is similar to the MSDN Library

http://blogdoc.biztalk247.com/default.aspx

My profile on BizTalk Blog Docs is: http://blogdoc.biztalk247.com/michael-stephenson

posted @ Tuesday, April 07, 2009 11:01 PM | Feedback (0) |

Message Archiving Pipeline Component and Windows Azure

By Michael Stephenson

I've been wanting to play around with some of the Windows Azure features for a while and following Steve Marx presentation at the MVP Summit and the release of the Windows Azure SDK (March CTP) I decided to give it a go.

In a previous post I had discussed how in the future versions of BizTalk I think they should begin to look at how BizTalk should look to take advantage of some of the newer technologies within the product itself. One of my suggestions was around taking advantage of cloud storage as an option for HAT data.

This post isn't really intended to go into the details of my thoughts on this, but for my experiment with Azure blob storage I decided to look at storing copies of BizTalk messages in the cloud.

One of my old colleagues (Nick Heppleson) manages the Message Archiving Pipeline Component project on Codeplex so my idea was to look at modifying his pipeline component to use the cloud for storage rather than the file system. I could then send this on to Nick so he can consider it as a future enhancement for that project. In addition to this I intended to run my sample in BizTalk 2009. The rest of this post will discuss the steps I took.

Upgrading the Message Archiving Pipeline Component

The first step was to download the message archiving pipeline component. I then setup the source code from the codeplex project on my BizTalk 2009 VPC using the BizTalk 2009 CTP. I then opened the solution and let Visual Studio 2008 do the upgrade of the solution.

This was very straightforward and did not have any problems.

Nicks code base contains a library folder containing copies of a couple of the normal dll's you would reference when creating a pipeline component. I decided that I would at first not update these to the BizTalk 2009 versions of those assemblies and only make the change if I found problems. I went with the assumption that when deployed this will just pick up the newer versions from the GAC and it worked fine indicating there are not any breaking changes in these assemblies.

It would probably have been a better idea to have updated these but in this example I wanted to make as few changes as possible.

Adding the Azure Client Storage to the project

At this point I had the solution open and ready, I now copied the C# project from the Azure SDK samples for the ClientStorage project. Without going into too much detail this project would abstract me from the REST interface and give me an easy to use object model to interact with Windows Azure.

My intention is to use the blob storage features of Windows Azure.

Making Changes to the Pipeline Component

I now needed to make some changes to Nick's pipeline component so it would have the right information to interact with the cloud. I took the following actions:

1. Firstly I removed the pipeline components properties and replaced them with properties so I can pass information from the BizTalk bindings to the pipeline component. The properties I needed are:
   1. The url for blob storage
   2. The account name for my Windows Azure project
   3. The secret key to get permission to my Azure project
   4. The container name where I would save messages to

       

2. I changed the IPersistPropertyBag implementation to load and save my new properties
3. I removed the file system specific code in the StreamOnReadEvent method
4. I removed some of the additional logic for macros and other advanced features that the pipeline component supports. I wanted to make the component a little simpler for the purposes of this demo.

You can see in the below picture the modified Execute method that I continue to use the CForwardOnlyEventingReadStream that Nick originally used. This means that the message will be archived in a streaming fashion which fits well with good practices for pipeline component development.

I then implemented my new logic into the StreamOnReadEvent method which would save the message to cloud storage. You can see in the below picture how as the stream is read I will upload the information from the buffer into the cloud via the StorageClient API. I also have logic to workout if I need to create or update the blob.

Setting up the Azure Project

To be honest there isn't really that much to setting up the Azure project. Once you have registered and signed in to the Windows Azure developer Portal on the following link http://www.microsoft.com/azure/signin.mspx you can proceed to the add new project button. On the below diagram I have shown the type of project you should setup.

Once you are viewing the project you can see in the below picture that it provides you with the end points and keys to access your storage - obviously I have changed the key since posting this J

Your Azure project is now setup and ready to begin accepting data.

Setting up the BizTalk Solution

In order to use the pipeline component I setup a very simple BizTalk implementation. I'm not going to go into too much detail about this but the implementation is summarised below:

• I will have a BizTalk Application with a receive location which will pick up a file from a folder
• There will be a send port which will subscribe to the receive port and push messages out to another file
• On the receive port I will setup a custom pipeline which will contain the Message Archiving Component in the decode stage of the receive pipeline
• I will configure the bindings to have the pipeline configuration setup using per instance pipeline component configuration. The below picture shows the configuration

In the picture you can see the Azure settings. One key point to note here is your project name needs to be provided in lower case. And the same with the container name (I think the container name also cant contain numbers or special characters)

Testing the result

Testing the process is simple. Drop a file into the input folder and it will come out at the output folder. The next question is how do I see what is in the cloud?

I chose to use the Azure Explorer project from codeplex to let me examine what is in my cloud containers. In the below picture you can see the following:

• Left Panel = The containers I have created
• Top Right Panel = The blobs that have been uploaded
• Bottom right Panel = The content of the blob I have chosen to display

Summary

Hopefully from this post you can see that it was relatively simple to start taking advantage of some of the Windows Azure features to compliment a BizTalk solution. While there are still a lot of hurdles to be overcome before the cloud is fully commercial and available to use it is good to think about the things this new paradigm offers us as solution developers and the benefits it can offer to solve some of the common problems we face.

In this post you can also see how simple it was to modify Nick's already excellent component to support this and hopefully it will be something Nick may offer in future versions. In terms of a design decision I certainly do not think that you should archiving to the cloud in all cases. There is the additional streaming overhead in doing this etc, but I feel that in certain circumstances it offers an interesting new choice. Remember that in the current message tracking to HAT there is enough different requirements and considerations to create a place for Nick's component in the first place. The additional benefits that storage to the cloud would offer is not having to worry about running out of storage or to have an archiving or backup policy. You would potentially just pay a cost based on the amount of storage you require.

Back at the start of the post my initial reason for suggesting that HAT could benefit from storage to the cloud would be that HAT essentially offers the option to do the actual archiving outside of the normal BizTalk process where as my modification to the Message Archiving Pipeline Component will be streaming this to the cloud as the service instance streams towards the messagebox. This obviously has potential issues around the connectivity to the internet and what to do if you have problems with the upload.

Anyway hopefully this provides some ideas for more in this area.

The modified code for this sample can be downloaded from the following location, but hopefully once Nick has looked over a copy of this he can possibly keep the sample on the codeplex page.

The code for this sample can currently be downloaded from the following link:

http://www.box.net/shared/0t5o39tddp

posted @ Thursday, April 02, 2009 1:10 AM | Feedback (0) |

BizUnit Step to Query HAT

I was just refactoring an orchestration today and to help me test it I wanted to query HAT to ensure the orchestration had completed successfully, BizUnit doesn't have a build in step to do this so I created my own which is easy to do thanks to the extensibility of BizUnit. I guess I could probably have used one of the BizUnit database steps and written some fiddly SQL to do this, but a new task would make this fairly reusable.

The xml to use my step is below:

<TestCase>

    <TestSetup>

</TestSetup>

    <TestExecution>

        <TestStep assemblyPath="Acme.BizTalk.Testing.dll" typeName="Acme.BizTalk.Testing.BizUnit.HAT.OrchestrationCompletedQuery">

            <DurationToCheckSeconds>120</DurationToCheckSeconds>

            <ExpectedNoOrchestrations>1</ExpectedNoOrchestrations>

            <OrchestrationName>MyOrchestration</OrchestrationName>

            <FailIfLess>true</FailIfLess>

            <FailIfMore>false</FailIfMore>

            <HATConnectionString>server=.;database=biztalkdtadb;integrated security=sspi;</HATConnectionString>            

        </TestStep>        

    </TestExecution>

    <TestCleanup>

</TestCleanup>

</TestCase>

Some key points to this XML are:

• The DurationToCheckSeconds is as it says a period of time that HAT will be checked for to see if your orchestration has completed
• The ExpectedNoOrchestrations allows you to indicate how many instances you expect to find from this query
• OrchestrationName is the name of the orchestration you are looking for
• FailIfLess allows you to indicate if an exception should be thrown by the step if it finds less than the expected number of orchestrations
• FailIfMore allows you to indicate if an exception should be thrown by the step if it finds more than the expected number of orchestrations
• HATConnectionString is the connection to the HAT database

In my step I will execute a query which will check for instances which started after the start of my BizUnit test (the step gets this from the BizUnit context).

In the example above im basically saying that I want to find atleast 1 instance of my orchestration completing since my test began.

As we all know HAT data sometimes takes a little while to be available. In my step depending on how you configure the settings above it will not necessarily wait the whole duration to check before confirming that the test was ok. For example In my usage above it lets me confirm that at least 1 instance is in HAT since my test begun, but I don't want to wait for ages ensuring that only 1 instance is there. Because my tests all run serially I can assume that so long as I find 1 instance then my test is good and I can continue as soon as I find it.

The code for the BizUnit step is available to download at the following link:

http://www.box.net/shared/647eo262x3

HTH

Mike

posted @ Monday, March 23, 2009 6:11 PM | Feedback (0) |

Thoughts about BizTalk VNext

Recently a few of us were speaking about what would be the future of BizTalk after the 2009 release. I already had a few thoughts on what I think would be great additions to the product and what strategic directions it may move in. I have checked to ensure that none of my thoughts breach the MVP NDA and after getting the thumbs up I've listed my thoughts below so there might be a bit of discussion on these ideas.

posted @ Tuesday, March 17, 2009 3:44 PM | Feedback (5) |

SBUG Meeting 26th May 2009

There is a UK SOA/BPM User Group event planned for 26th May 2009

For details please refer to http://sbug.org.uk/forums/p/122/174.aspx#174

posted @ Monday, March 16, 2009 10:52 PM | Feedback (0) |

SBUG Event 15 April 2009

There is a UK SOA/BPM User Group Meeting planned for the 15th April 2009.

For details of the event please refer to; http://sbug.org.uk/forums/p/121/173.aspx#173

posted @ Monday, March 16, 2009 10:51 PM | Feedback (0) |

My First Custom WCF Lob Adapter Walk Through

Article Source: http://geekswithblogs.net/michaelstephenson/Default.aspx

I've been wanting to take a deeper look at the WCF LOB SDK for a while. I've read a few things about it, but you can't beat just getting stuck in and writing some code to figure out how it all works. My aims from this exercise were as follows:

1. Start to get an understanding on how to write custom WCF Lob adapters
2. Do something that has some relation to the real world and not just a hello world thing
3. Identify how much further I want to take this subject in terms of exploring its capabilities
4. Try to keep this first attempt simple to grasp the basics and maybe do more later

To begin with I had read a few things but quickly it became apparent there isn't a simple walk through available on the process you go through to create a custom WCF Lob adapter. There is lots of information on the background of the SDK and there are a few samples which show a fully working adapter. I'm hoping that this article will help plug this gap.

I'm not going to go much into the background of the SDK as there are a number of existing resources which are very good and I will mention some of the ones I have used later in this article.

Setting the Scene

As mentioned already I didn't want to just do the old Hello World sample. I wanted to do something which could have a real application so I would think about some of the design considerations you might come across when you are thinking of a Custom WCF Lob adapter. The scenario I chose to use was one from a project I have been involved with. I think I have blogged previously about a situation we had on this project where we had an old application which we wanted to integrate with. The application (which I will refer to as the socket application for this post) is exposed as a socket interface. On this project we had done a POC at one point to explore using the BizTalk TCPIP Socket Adapter to connect with it. While functionally this adapter worked fine we found that under load we would encounter some problems and strange errors. We ended up with a solution where we wrote a web service façade in C# which encapsulated some C# code to do this socket interaction (I think we were able to reuse some existing code which did this). We then deployed this web service to the BizTalk servers and made BizTalk do a call to the web service to interact with the socket application. The results proved that this web service although there was the additional overhead, the latency was only slightly longer but the performance when under load was significantly better and the service was much more reliable than the custom adapter. The architecture looked like the below diagram.

At this point before I got into my custom adapter there were a few thoughts I wanted to consider.

1. Why can't I just use the netTcp Binding?

For completeness I did consider this and it is common that with non WCF applications if you try to use the NetTcp Binding you may get problems because the binding is not intended to be interoperable and is specifically for WCF. In the case of this application I think I got a .net Framing exception. There is a forum post which discusses this topic and discussed creating a custom binding to handle this.

1. Can I reuse our custom code within the adapter

I don't particularly want to go writing a bunch of socket code so my preference was to reuse our classes which had been implemented to encapsulate this code. I wanted to be able to plug them into the adapter in a tidy way.

1. Would the adapter be an application specific adapter or a protocol adapter

I was considering if the adapter I would create would be specific to connecting to just this socket application or if it would be more generic and able to connect to most applications which support a socket connection. I had a look at our existing code and while it looks fairly generic I think there is a small bit of (dodgy poorly placed) code that does something specific to this application. I have decided to consider that this particular thing is really small and I will essentially ignore that for now and just consider the adapter to be a protocol adapter. It will essentially be a socket client adapter. The socket client class we will wrap has a bunch of settings which can be configured and I intend to make these configurable as a WCF binding element so the adapter could be used differently when consumed.

Before I go through the walk through I think its important to list a few things that the adapter will and wont do in relation to the WCF Lob SDK just to set your expectations correctly:

• The adapter will be a one way synchronous outbound adapter
• It will not handle inbound messages
• It will not perform asynchronous actions
• It will not perform meta data look ups

Most of the limitations above are mainly because this fits with the scenario above and it also allows me to keep the scope of this POC quite small and specific. Often I find if you try to do too much you can end up not making good progress. It will also give me future scope to do more posts about these other things.

Prerequisites

I developed this sample on a BizTalk 2006 R2 developer machine to which I added the following components:

• WCF LOB SDK SP1
• Windows Platform SDK

The Walk Through

To begin with I created a Visual Studio solution and added a windows forms application to test the adapter and a console application which would act as a stub of the socket server application.

The Socket Server Application

The socket server application is a console application which will when executed sit listening on the configured port for any messages and respond with a preconfigured response. The below picture shows a sample of the response message which is a delimited string.

Creating the Lob Adapter

This section will describe how I created the adapter itself.

Generating the Adapter Project

To begin with I chose to add a new project to the solution and picked the WCF LOB Adapter project type. I called the project SocketAdapter.

Next the WCF Lob Adapter Wizard opens and once past the welcome screen I was able to provide some information about my adapter like in the below picture. I supplied the scheme which I decided to be netSocket and also a namespace.

On the next screen I am able to choose the features my adapter will support. As mentioned earlier this adapter will be quite basic and will support only synchronous outbound messaging and will not support any meta data features. This can be seen in the below picture.

On the next screen I am able to supply some properties for my adapter. I have chosen that the properties will be the Maximum Buffer Size which will control the retrieval of data and the Protocol Type which will configure the socket client to determine which protocol it will use. While the Protocol type is a string property I will ensure in my adapter it is of type System.Net.Sockets.ProtocolType and I will mark the default value in this wizard to be Tcp.

The next screen allows you to configure the parts of the connection used by your adapter. I have determined my connection will need to include a server and a port.

The next screen provides a summary of your chosen options like in the below picture

When I finished the wizard the code for my adapter was generated. The below picture shows the code files which within my new project.

Adding the Custom Code to My Adapter

The next steps were to apply the manual steps including adding code to the adapter to turn this boiler plate code into a functioning adapter.

Strong Name

The first thing I did was to create a strong name key and amend my project so that it is strong named when it is compiled. I'm going to assume that you already know how to do this.

Amending the Adapter Project File

I wanted to amend the MsBuild for this C# project so that when it was compiled I would over ride the AfterBuild event and register the adapter in the GAC. To do this I added an xml file to the project called OverrideBuild.xml. The below picture shows this.

The next step is to amend the MsBuild xml in the .csproj file so that it will import my custom build target. The following picture shows this edited version of the SocketAdapter.csproj file.

Modify the SocketAdapterConnectionUri Class

One of the classes I needed to add code to is the class that manages the Uri for the adapter. The first thing I did was to amend the Uri property so that my class would correctly parse and create a Uri object based on my connection details. A well formed uri would be something like netSocket://{MachineName}:{PortNumber}

I also needed to modify the constructor of this class so that it would save the details of the passed in Uri when it was constructed.

Adding the Custom Socket Client Class

The next step was to copy across the couple of classes that get used with the SocketClient class. I did not amend these classes just copied them over as is into the adapter project. Remember from earlier the SocketClient class performs all of the stuff to interact with a socket and send a message to the socket application and receive a response. This is the class that we have already used elsewhere and which I want to just plug into the WCF Lob SDK framework to create an adapter. The below picture shows the outline of the classes.

Modifying the SocketAdapterConnectionFactory

I needed to make a couple of changes which are highlighted below in the picture to this class. Basically the change is to make the connection uri properties available to the connection object once it has been created by the factory.

Modifying the SocketConnection Class

The next step was to modify the SocketConnection class. I implemented the following changes.

Add a Field for the SocketClient

This field will store a reference to the SocketClient object. The below picture shows this.

Expose the SocketClient as a Property

The below picture shows how the SocketClient field was exposed as a property so it can be accessed by the outbound handler.

Update the Constructor

The below picture shows how the constructor was modified so that the SocketClient instance is created. The properties are extracted from the SocketConnectionUri and the binding information to configure the SocketClientArgs. One thing to notice in the below picture however is that I have missed a bit where I have not extracted the timeout values from the binding information and have still got them hard coded. You should look to get these from the binding info.

Modify the Implementation of IConnection

The below picture shows how I changed the implementation of some of the methods for this interface. In these methods I open and close the SocketClient connection.

Modifying the SocketAdapterOutboundHandler Class

The final class within the adapter to modify is the SocketAdapterOutboundHandler. This is the class which will send a message and handle a response. The below picture shows the implementation of this method.

There are a couple of assumptions that this method makes:

1. The inbound message action will be "Submit"
2. The inbound xml will have 1 element within the body and the content of that element is the data to send to the socket application. This essentially means we will need a contract with has 1 input parameter of type string.
3. The outbound xml will be in the namespace defined in the code
4. The outbound message will look something like:

<SendResponse xmlns:http://acme.wcf.lob.paymentapplication/>

    <SendResult>{Value goes here}</SendResult>

</SendResponse>

Now all of the adapter code changes are complete.

Deploying the Adapter

The steps to deploy a WCF Lob Adapter are described in the MSDN documentation. Since we have already strong names and Gac'd the assembly the only thing left to do is to register the adapter in the machine.config to make it available to applications and services. This is also described in the above linked MSDN documentation, however the steps I took are as follows.

After opening the SvcConfigEditor.exe tool I browsed to the machine.config file and opened it with the Service Configuration Editor. I then selected the Binding Extensions node highlighted in the picture below.

The next step was to click the New button and then provide a name for my binding extension and choose the assembly and type from the GAC as shown below.

Once I had done this you can see in the below picture that my custom WCF Lob adapter is now registered.

The Client Application

Finally the adapter is available for use. In this demo I have chosen to keep things simple and to consume this adapter from a .net client rather than from BizTalk. In a future post I might extend this demo and use BizTalk. The steps I did to make the windows forms client be able to consume the netSocket custom WCF Lob adapter were as follows.

Defining a Contract

The first thing to do was to define a contract. I think with most custom WCF Lob Adapters if you implement the meta data features then you should be able to generate a contract, however in this demo I didn't do that so needed to create a contract manually.

Remember the bit above where I implemented the OutBound Handler in the adapter I listed some assumptions about the message I would receive. Well those assumptions basically equate to a contract like I have defined below. The contract is a method which has input and return parameters which are both strings. Note the namespace and action properties also need to correlate.

Adding Code to the Form

The below picture shows the code I added to the form. You can see this code is really simple and does nothing more than the normal WCF style code for consuming a service.

Configuring the Client

The final step was to configure the client to be able to call the service. To do this I opened the App.config file using the Service Configuration Tool which I used above. This time I added a client endpoint which was a netSocket binding. See below picture.

The next step was to add a binding element for the netSocket adapter. This allows me to specify the properties for the binding we discussed much earlier such as MaximumBufferSize and ProtocolType.

I then supplied the settings for the client endpoint. You can see these below.

Once I saved the configuration I can look at the App.config file and would see the below xml. You can see that some of the properties such as ProtocolType aren't configured because they are just using their default setting.

Running the Demo

We are now complete with development and can run the demo. Before we hit F5 we need to configure the solution so that it will start the socket server and client projects both at the same time. For those who may not know the below picture shows how to do this.

When we hit F5 both applications will pop up like below and we are ready to test.

Clicking the test button makes the client perform a call via the netSocket adapter to the socket server. You can see below the console application recorded that the server received a message, and the client popped up a message box displaying the response.

At this point you can see we have successfully made the call.

POC Summary

In summary I quite enjoyed this POC it was fun to play around with the WCF LOB SDK. Some of my thoughts after the POC are as follows:

1. Hopefully the walk through shows what needs to be done to make a simple adapter work. Its not as difficult as you might first think when you generate the adapter project and then cant workout where to put your custom code.
2. One of the things I didn't quite get my head around is the relationship between a contract and how the adapter handles different types of messages. I think part of this was because I chose not to implement the meta data features. My assumption is my client could have generated the contract and it would have made more sense.
3. It proved to be a pain writing the code to parse and create the response messages. I think some of this may relate to the above point, but I think it would be good to have some samples of best practices for doing this.
4. I think I will look to consume this from BizTalk also just to see if there is much more required to do that
5. It proved to be very easy to reuse our existing SocketClient code within the WCF LOB SDK framework without having to make any changes to the existing code

My final thought going back to the original problem domain I was looking at is that I do not think this WCF Adapter necessarily replaces the pattern we created originally where we created a façade service on top of the socket application. While this adapter has its uses and functionally can do the job some other architectural considerations from our original problem domain are not solved.

One of the issues with the socket application was that it was consumed from a number of applications meaning that it was widely available. The application also didn't implement a strong security mechanism so we had a vulnerability here. To address this when we implemented the façade web service we secured this by restricting it to be only able to be called by users in certain windows groups and we signed and encrypted the message on the wire and passed identity within a Kerberos token. This allowed us to lock down the socket server with IPSec so that it could only be accessed from certain machines. The "other applications" would then either consume the web service directly or call BizTalk which would handle some of the security and transport requirements to allow the application to integrate with the socket application via the façade web service as in the diagram right back at the start of the article. At this point we have mitigated a lot of this security risk.

If we allowed all of the clients to call the socket application with the netSocket adapter then it would require us to open up this security hole again. I think these considerations make it clear that implementing the service façade has a lot of value in abstracting the consuming applications from the socket application, however where the WCF Lob adapter could have a place would either be behind the service façade so that the web service uses the adapter to communicate with the socket application or if it was used by BizTalk and the service façade logic was moved into BizTalk which then used the WCF Adapter (Note: on this point we would need to ensure that the original performance issue didn't exist that we had with the BizTalk TCPIP Adapter).

I hope this article provides a lot of food for thought and helps people get started with the WCF LOB SDK and I'm sure ill post more on this soon.

The sample code for this article is available at the following place: http://www.box.net/shared/7xsvoe12zi

posted @ Tuesday, February 10, 2009 2:04 AM | Feedback (6) |

Good Article

I came across this article by Paolo and thought it was worth pointing out to those who might not have seen it as it is really useful

http://blogs.msdn.com/paolos/archive/2009/01/20/biztalk-server-and-protocol-transition.aspx

Keep up the good work Paolo

posted @ Thursday, February 05, 2009 8:18 PM | Feedback (0) |

SBUG Newsletter

We have just published the 1st newsletter from the SOA/BPM User Group.  It is available to download here:

http://sbug.org.uk/media/p/157.aspx

posted @ Thursday, February 05, 2009 8:56 AM | Feedback (0) |

Reconfigure BizTalk with MsBuild

The other day I was having an issue with our build server so I decided to write a script to reconfigure BizTalk on our build or development servers. My plan was to run this regularly on a scheduled basis to ensure our build servers were kept clean.

I thought id write a little about this as it may be useful to others. Note the script is aimed at a single machine hosting BizTalk 2006 R2 and SQL Server.

Before I started I exported the configuration of our BizTalk Group to a file and then amended the credentials etc.

The list of tasks I would perform to clean and reconfigure our server is as follows:

To implement this I created a Visual Studio solution and wrote an MsBuild script and used some of the Microsoft.Sdc tasks. The below picture shows the files in this solution.

Its pretty simple and I will provide a link to the sample at the end of this post.

Some of the other ways I have extended this script since include:

• Using an MsBuild task to register some custom adapters
• Using an MsBuild task to register the WSE2 and Enterprise LOB adapters
• Using an MsBuild task to add adapter handlers
• Using the MsBuild configuration dictionary stuff I have blogged about in previous posts to configure the BizTalk Configuration xml file for different environments

The link to the sample for this post is: http://www.box.net/shared/t4k0m6ps5y

posted @ Wednesday, February 04, 2009 11:55 PM | Feedback (1) |

Implementing the Resequencer Pattern with BizTalk 2009 and Frends

Article Source: http://geekswithblogs.net/michaelstephenson

I've recently been looking at a proof of concept of implementing the resequencer pattern in BizTalk 2009. When I've implemented this pattern in the past it usually required a lot of custom development to create a database to support the resequencing process. If you would like to see more on the theory of the resequencer pattern then refer to the following link: http://www.eaipatterns.com/Resequencer.html.

I've seen a few .net and BizTalk examples which discuss doing the resequencing in memory, but for my case I felt this would be a limiting factor as it would mean I could only use 1 machine to implement this on and there could be scalability issues. The aims of my proof of concept were as follows:

• To create a demonstration of the resequencer pattern that would be scalable to handle lots of messages in BizTalk
• Be able to extend this relatively easily to handle resequencing messages from multiple sources. For example if you had many partners pushing in messages and you only had to resequence messages from each partner separately
• Minimise custom development and maintenance requirements to support this. I didn't want to have any custom databases in my solution.
• Try to make the resequencing pattern implementation as reusable as possible

Before I got into the details of this I was already familiar with the Frends product which I have discussed how it could be used to compliment BizTalk solutions in previous articles. My thought was that I might be able to use the configuration store part of Frends to help me to manage the sequencing in my POC. The rest of this article will discuss how my POC works.

My POC Environment

The environment on which I have build my POC is a Windows 2008 virtual PC running the CTP of BizTalk 2009 and SQL Server 2008. I have also installed the trial version of Frends.

Configuring Frends

I would need to configure a "Connection Point" in Frends. A connection point is basically an instance of some configuration data. You can set these up through the Frends Management Console where you will provide a schema which will define your configuration data, and then in Frends you specify values for the schema elements. To do this I took the following actions:

1. Create Scheduling Configuration Schema

In my Visual Studio solution I created the below schema. In this schema I have defined an element to hold the sequence number of the last message processed, and an element to hold the time the last message was processed.

1. Create Connection Point

In the Frends Management Console I would choose to add a new Connection Point like in the below picture.

On the add new Connection Point screen you need to supply the name of the connection point and choose a schema which will represent the configuration data. I have chosen the schema I created above and you can see in the below diagram that it will create a designer to allow you to supply values for the configuration. To begin with I will leave the time element blank, but I will supply the value zero to be the last processed message as it will indicate the start of the sequence. One interesting thing about the connection point is that you can have multiple connection points with the same name if you supply a different Party Name for each. This lets you correlate a specific connection point instance to a party you may be dealing with.

In the picture below you can see my configured Connection Point.

Setting up my BizTalk Solution

In setting up my BizTalk solution I have two projects. The first project will contain my BizTalk artefacts and the second will contain some utility components. These are shown in the below picture.

The key points of the solution are described in the below table:

The input message will look like the below picture. It simply is a message containing the number of the message.

For each message that is received an instance of the SequencedOrchestration will start. The below picture shows the orchestration and I have highlighted some of the key points to discuss:

1. The loop will continue until the variable which indicates this instance is the next in sequence is set to true
2. This expression shape will call the SequencingFacade to work out if this instance is next
3. If this instance is not next then the instance will delay for a while
4. When this instance is next the loop exists and I will create the message to send out. I also set the File Adapter context property do I can out put a file name which is the same as the sequence number
5. This expression shape will call the SequencingFacade to update the last processed sequence number

The Sequencing Façade

In the SequencingFacade class the first method to discuss is the AmINext method. This method will call Frends to obtain the connection point information. Frends makes the Connection Point subsystem available as a web service using WCF. Once I have called the web service I can get the configuration data from the response as a string of XML. Because I'm using BizTalk 2009 and can compile the .net 3.5 features in my solution I have chosen to use Linq to XML to parse this XML to get the sequencing details.

I can then implement some simple logic to work out if the instance is the next in sequence which you can see in the below picture.

The next method in the SequencingFacade is the UpdateSequence method. In the below picture you can see I get the sequence information from Frends and then use Linq to XML to update the xml to indicate a message has been processed. I finally call the Frends web service to update the connection point.

Walk Through

Now that we have reviewed the highlights of this implementation I will walk through a sample of how it works.

1. We are currently at the start of the process and I have configured Frends that the last message processed was zero. The output folder contains no messages and there are no instances running in BizTalk.
2. I add message with sequence 1 into the input folder. This message is collected and processed. Message 1 ends up in the output folder, and the Frends connection point is updated as in the picture below

1. I now add message 3 to the folder. You will that this instance will be running in BizTalk and will keep checking Frends to see if its next, and will keep using the Delay shape between checks.

1. I will now add message 4 to the folder. You will see this will also continue waiting like message 3.

1. I finally will add message 2 to the input folder. You will see that it will process first and complete, then message 3 and 4 will follow it out in order. You will be able to see all messages in the output folder and finally Frends will know that the last message processed was 4

1. I also has debugview running during the walkthrough and you can see from some of the trace messages I recorded that the messages were processed as expected

    

   

    

    

Summary

My proof of concept proved to be quite successful. I was able to setup this demonstration very quickly and it did everything I required. I think I have been able to show an implementation for the resequencer pattern which will be quite scalable and able to handle a lot of messages by taking advantage of the BizTalk scalability model and its ability to process lots of orchestrations at the same time. Some of my other thoughts about this pattern which may help move this to a real world implementation are as follows:

• I would be looking to use the party name bit of Frends to help me handle messages coming from different sources
• I would probably look to use something like NCache or Velocity to do a distributed caching of the responses from Frends if I was going to have a high load scenario. I need to have the permenant storage which Frends provides but by caching the sequence number and updating the cache when a process is complete I would not need to call the Frends web services very often. Also using one of the distributed caching solution would mean I can synchronise the cache across multiple machines within my BizTalk group
• I have coded the interaction with Frends in C#, but it could equally be done in an orchestration by calling the services with a send port etc.  The Frends service would normally be running on your BizTalk machine anyway so I guess you could choose how you did this
• I could probably implement the whole loop in a seperate orchestration where you would pass in the name of the connection point (and optional party) and your sequence number to this orchestration which would then interact with Frends.  This means you could potentially implement resequencing of any of your orchestrations by just calling this other orchestration (note this will mean you have more running instances)

If you would like to have a look at the sample code then it is available on box.net at the below link. (Note some companies block box.net so if you are unable to get the sample then let me know)

http://www.box.net/shared/ircluappxe

posted @ Thursday, January 01, 2009 11:11 PM | Feedback (0) |

Refactoring a common anti-pattern when publishing web services from BizTalk

Background

I was recently reviewing a BizTalk application where there were around a dozen orchestrations which had been exposed as either SOAP or WSE web services. The typical pattern that had been used in these orchestrations is illustrated in the below picture. You can see that there is a Request/Response port and in the orchestration initialises based of the request message then returns a very simple acknowledgement type message (just a message of .net type System.Int32 with a value of 0) to indicate the orchestration has the message and is going to do some additional work.

This is a pattern I've seen implemented particularly with inexperienced teams. Often when I've seen this pattern the code also had poor testing and the team did not have much confidence in their implementation. They ended up with this pattern to ensure that the caller knew they had the message. The main issue the customer had was that occasionally they would experience a timeout at the client end particularly during load, it wasn't a major issue as it only happened occasionally and the client could recover from this ok, but although the client got a timeout the orchestration still processed anyway and would make the appropriate updates to the original application to update it on the progress.

The Challenge

The challenge in this refactoring exercise was as follows:

• The application was already in use so we are not able to change any of the contracts used by client applications. This means the WSDL for the published WSE and SOAP web services can not change.
• We wanted to reduce the number of receive instances waiting around for a meaningless response and as result reduce the load a little

Before we get into the details of the refactoring, we will first look at the code samples from the previously generated SOAP and WSE services.

Existing WSE Web Service

In the below code snippet you can see that firstly I've had to censor some of the code from this article, but secondly it's a standard web service generated with the WSE adapter. It inherits from the WSEReceiver class and invokes the receive method and indicates that it's a 2 way call, then returns the response.

Existing SOAP Web Service

The below picture shows a sample of one of the SOAP web services generated from an orchestration. Again pretty standard generated web service using the SOAP generating wizard.

Additional Notes

In addition to what I've mentioned above, some other things which affect this refactoring are:

• The web services are generated every time we run a build and they are also versioned

The Refactoring

The following sections will describe how we refactored the solution to implement our desired improvements.

Manually Implementing the Web Service Projects

The first thing we needed to do was to change the web services from being automatically generated to being two projects within the solution. We creates these two projects but with the request and response types being generated from schemas within the solution we wanted to maintain the ability to update these web services automatically so that any changes to schema were reflected in the web services. In the msbuild for the new web services projects we extended the build process so that C# classes were generated in the web projects representing the schemas using xsd.exe. In the below picture you can see we are generating these files into a folder called Contract within the web services.

After this we basically copied the artefacts from the generated web services to the manually build web service projects. We also needed to ensure the correct references were added.

Changing the SOAP Web Service

In each of the web services we needed to make a few small changes from what was generated. The first change (the top red circle) was to change how we assign the bodyTypeAssemblyQualifiedName variable. In the generated web services this is hardcoded by inspecting the assembly, but when we want to do this manually I added a reference to the schemas assembly and then just used the highlighted code snippet to get that name.

The second change (the bottom red circle) is the change to the call into the BizTalk assemblies where we change the oneway parameter to indicate that we are not waiting for a response. Previously the zero response message came from the orchestration meaning the messages route had been into the message box, into an orchestration, back into the messagebox and then finally back to the waiting instance to reply to the caller. By doing this as a one way call the code will basically wait until the message has been persisted to the messagebox and then continue.

You can see I then return my own integer of value 0 to indicate it has worked.

Note I've also implemented some custom logging to log any errors.

Changing the WSE Web Service

The WSE Web service changes are very similar to the soap ones. In the picture you can again see the change to how the schema qualified name is obtained. We have also changed the oneway parameter when we call the invoke method (although I've missed this off the right of the screen shot)

Changing the Orchestration

Now we are managing the web services as projects within our solution we need to change the orchestrations. The picture below shows we now have a one way receive port which is no longer late bound like previously, but is now direct bound based on the message type. By the time we have initiated the orchestration the caller already knows its request hit the messagebox and it does not need to wait around for confirmation that the orchestration has started.

Changing the bindings

We finally needed to update our binding file template to change the receive locations to be one way.

Testing

We had already implemented a number of BizUnit tests and were able to use these to confirm everything that had worked before the refactoring still worked exactly as it did now and that none of the web service contracts had changed.

Summary

In this post we discussed a common anti-pattern you seen implemented with BizTalk. We were able to take a pragmatic approach to addressing some of the issues the customer was facing with this implementation while minimising the affects to the consumers of services published from this application.

posted @ Monday, December 29, 2008 3:02 PM | Feedback (3) |

Dynamic BizTalk Versioning Strategy with Cruise Control

Article Source: http://geekswithblogs.net/michaelstephenson

Versioning is one of the more difficult aspects of a BizTalk project mainly because of the ability to deploy side by side versions of the components within a BizTalk application and the fact that often a BizTalk project would require long running processes making it more difficult to do deployments. There are a couple of useful resources available on this area:

• In his article Versioning and Re-Deploying BizTalk Server 2006 Applications Richard discusses some different scenarios about when you deploy side by side versions of BizTalk and makes some interesting observations. If you are new to versioning for BizTalk then this is good background reading.

   

• In the MSDN documentation for BizTalk there is a page which discusses versioning and the 2 main strategies for it. This is useful but from my experience I think it misses a couple of things and is wrong in relation to the BAM files

   

• This web cast includes a discussion around BizTalk versioning (mainly focusing on how to partition your BizTalk artefacts)

   

• Nick Heppleson wrote an article about how you can use assembly redirection with BizTalk

   

• There is also a new article on MSDN which provides some information similar to this article called Developing Integration Solutions using BizTalk Server 2006 and Team Foundation Server. Although again they explain how you could do versioning but don't seem to walk through their approach.

   

While all of these articles are great there is a gap in that nothing really talks about how to implement a versioning approach in your development process. In the rest of this article I will discuss some of the different considerations and how we implemented dynamic versioning for our BizTalk projects at my current project. I will also provide links to a sample which shows this.

Considerations for Versioning with BizTalk

There are a number of different things you should consider for your versioning strategy which are discussed below.

Type of Versioning

There are two main types of versioning which are discussed in the MSDN article above:

• Static Versioning

In static versioning you would normally keep the Assembly Version Number the same all of the time and only increment the Assembly File Version with each build. This is the most common type of versioning approach I have seen used.

• Dynamic Versioning

In dynamic versioning you would increment both the Assembly Version and Assembly File Version with each build. The most common way I have seen this implemented is to manually change these version numbers (this subsequently requires a number of changes to files within the project)

Deployment Pattern

The way you plan to deploy your application can have an affect on the versioning approach you might use.

• Replace application every time

Some BizTalk applications implement processes where there are clear windows where a deployment can be done. There are no significantly long running processes and it is easy to drain the system of in process messages. In this case when you deploy a new version of the application you would be likely to completely remove the existing BizTalk application and then add the new version.

This approach is well suited to either of the 2 types of versioning.

• Upgrade application

If you have a BizTalk project where you can not get a clear deployment window and you can not drain the system then your deployment approach needs to be able to support running multiple versions of processes at the same time. In this case you are essentially applying updates to your existing application and when the older instances have completed you may remove the old assemblies.

Latest Version for Development

One of the challenges is how within your code base you will implement these side by side versions and develop/test against them. When we deploy an official version we will take those assemblies and add them to an external libraries folder within our code base. When we build the code we will also deploy these older versions of the code and then run tests against this combined solution. One key thing we do is that we version the BizTalk project files in the code base to be version 999.999.999.999. When we do a local developer build we do not change the version numbers we only do this on our build server when the build is ran by Cruise Control. We need to ensure that the assemblies built from the code will always be the latest version when doing a developer build so we chose this approach.

What types of files do I need to care about for versioning?

There are a number of different file types you need to care about when applying versioning. In my opinion the above article on MSDN doesn't cover everything it needs to so here are my thoughts below:

Binding Files

We already use an approach of tagging the binding files to support changing the configuration for different environments based on stuff I have discussed before in other blog posts. In a binding file any references to assemblies will typically use the strong name for that assembly. When we apply dynamic versioning in the development process we will need to update parts of the binding file in each build to ensure the binding file continues to be valid.

If this is not done correctly the we could get problems importing a binding file.

Web Service Description Files

If you publish any web services from your solution then you may choose to save the web service description files and then regenerate the web services as part of your build. We save these files and have MsBuild tasks to support regenerating these web services with each build for WCF/WSE2 and standard SOAP web services. When we apply version updates we need to amend the WebServiceDescription.xml file that is published before we do the build.

Map Files

BizTalk Map files may contain references to assemblies if you execute an external assembly using the scripting functoid. We need to ensure these maps are updated if the assembly version is changed. If we do not do this we may get errors while executing the map.

Pipeline Files

BizTalk Pipeline files may have references to external pipeline components or schemas or other assembly references in custom configuration. If we apply dynamic versioning we need to be aware of the impact on these files and make appropriate updates to ensure the pipeline still works when it is compiled.

BAM Files

If you use BAM on your project then your solution may contain Tracking Profiles or BAM Observation Models. The BAM Observation Model does not contain information relating to assemblies, but if you use any Tracking Profiles (.btt files) then they may contain references to assemblies which may need to change as you change the version number. In the MSDN documentation it indicates that .btt files are in binary format which Im not sure is right as the ones on our project are clearly XML files. Based on our experience you can just update these files to change the version numbers.

One side comment is that you can also use the Typed BAM API project on CodePlex which will mean you don't have to worry about Tracking Profiles.

BRE Policies

If you manage your BRE policy as an XML file which you import and export then you may also need to update version numbers in it if you reference assemblies.

Configuration Files

Some configuration files in your solution may have assembly references in them. You may need to update the version numbers here too.

Our Versioning MsBuild Tasks

As you have read so far there are a number of different tasks which need to be done to implement versioning. We have developed the following list of MsBuild tasks which help us to do this in our MsBuild process. Note the code for these tasks is available in the accompanying sample.

BizTalkProjectAssemblyInfo

This task will look for any .btproj files within your solution directory (including sub directories) and then update the version attributes of the project file to take the value from Cruise Control

AssemblyVersion

This task will create a file called VersionNumber.cs which contains the C# attributes for versioning an assembly. All of the C# projects within our solution reference this file and pick up its version number when they are built.

ApplyVersionToAdhocFiles

This task is given a list of files which need to be inspected for versions to be updated. The task will use the assembly name regular expression, assembly public key token, assembly culture and assembly version regular expression to identify strong names within the file that need to be updated. Once a strong name is found the version number part of it is updated to the version passed into the task by Cruise Control.

Note that this task is also used to update any BRE policies which are stored as XML files in the solution.

ApplyVersionToWebServiceDescription

This task does the same as the apply version to adhoc files task however this task needs to force the file to be rewritten in a the right encoding for the web service publishing wizard to use.

ApplyVersionToBindingFiles

This task is given a list of paths to binding files which it needs to update. In the binding files it will look for strong names in the same way as the adhoc files task and update them. In binding files there are also some other formats in which assembly version numbers are stored so this task will also handle these to ensure that the binding file is correctly updated.

ApplyVersionToBizTalkArtefacts

This task will start at the root directory of the solution and look for all BizTalk artefacts which may require updating (Maps, Pipelines, Tracking Profiles). For each of these file types it will look for assembly references based on the strong name patterns provided and apply the updates to the version numbers which come from Cruise Control.

Walk Through

Setting up a new solution to use dynamic versioning

When we are starting with a new solution and we want to be able to implement the dynamic versioning straightaway there are a couple of standards we use.

C# Project Files

In our solution it is common to have C# projects to compliment our BizTalk projects. We take the approach of versioning these assemblies by adding an existing file to the project which includes the below code snippet (we usually call this file VersionNumber.cs and it is located in the root folder of the solution). This means that the assemblies are all versioned to the standard 999.999.999.999 during development

When our MsBuild process executes it will overwrite this AssemblyVersion.cs file and replace the version numbers in it to the version label taken from cruise control. We use our AssemblyVersion custom MsBuild task. There are a number of different tasks in the community and Microsoft.Sdc tasks which can do versioning of C# projects and using any of these should be sufficient.

BizTalk project version numbers

On all of the BizTalk projects within the solution we will amend the project properties (see pic below) to set the version attributes to the fixed 999.999.999.999. This will mean during a local developer machine build all of the BizTalk assemblies will be versioned as 999.999.999.999. When Cruise Control runs a build the extra build tasks will run and execute our BizTalkProjectAssemblyInfo custom MsBuild task. This will update the version numbers of all of the BizTalk project files within our solution to apply the version number from Cruise Control.

Other Artefacts

As we will have applied the 999.999.999.999 version number to our projects before we start adding artefacts to the projects then any files we add will correctly pick up these versions and we will not manually need to update them.

The Cruise Control Integration Bit

In the below picture you can see the Cruise Control configuration used for the Continuous Integration of this solution. In the tasks bit you can see we basically call 3 MsBuild scripts as follows:

1. Apply versioning to the files in the solution
2. Perform a debug/development build of the solution (including executing tests)
3. Perform a release/deployment build of the solution (including executing tests)

    

The bit we have added is the first step where we will call ccnet.targets MsBuild script. This file is in the solution and is where we will call the tasks to apply versioning.

Setting up the CCNet.targets File

The below picture shows the contents of the ccnet.targets file. This file will basically call the list of MsBuild tasks in the ApplyVersion target which will apply versioning to all of the appropriate files. Each of these tasks was discussed in detail earlier in this article.

In the property group at the top of this file you can see that we have defined some common attributes which each of the tasks uses to do its job. These properties are used as follows:

As you can see from the order of calls to our tasks we version the files in the following order:

1. BizTalk project file
2. C# project file
3. Adhoc Files
4. Web Service Description Files
5. Binding Files
6. BizTalk Artefacts

The order here isn't really that important as long as all of these activities are done before compiling the solution.

Converting an existing solution to use dynamic versioning

This walk through will explain how to configure an existing BizTalk solution to use dynamic versioning rather than static version approaches described above. One key point to reiterate here is that on a developers machine we always do builds as our fixed 999.999.999.999 version number and it is only on the continuous integration build server that we run the additional tasks to do full versioning as described below.

Reconfiguring parts of the solution

C# & BizTalk Project Files

We will need to update the BizTalk and C# projects to reference the 999.999.999.999 version number. We can do this the same as how is described in the section about setting up a new solution for dynamic versioning.

Updating Map Files

Maps are slightly strange. In the map if you reference an external assembly and execute it via the scripting functoid then you need to be aware that the strong name for executing assembly may need to change. In the MSDN documentation explaining this it indicates you need to change maps, but if you disassemble an assembly containing maps and the C# project was referenced as a project assembly then the compiler seems to change the xslt within the map for you. However as this is a little unclear we decided to change the maps anyway.

To do this we needed to open the map using the XML Editor in Visual Studio. We then would change the version numbers highlighted below as 1.0.0.0 (in the pic below) to 999.999.999.999 for all of the assemblies we know we are going to dynamically version.

We would do this for every map in the solution. When Cruise Control runs the build it will modify and strong names in .btm files which need to be versioned.

Pipeline files

Pipelines are similar to maps in that under the .btp file is XML which may contain assembly strong names (see picture). Again we would open the file using the Visual Studio XML Editor and change the version numbers from 1.0.0.0 to 999.999.999.999 for any references to assemblies we are going to dynamically version.

.

When the build is ran by Cruise Control our additional tasks will find all of the details we need to change and modify the .btp file before it is compiled.

Binding file template

In the solution we have a template binding file which contains tags which we use to configure the binding file for different environments. In a binding file there are lots of references to assemblies by their strong name. If we have any assemblies which we know are going to be dynamically configured then we will replace their version with the fixed 999.999.999.999. In the below picture you can see an example binding file template which shows the version numbers before they have been amended.

When Cruise Control runs our build then an additional step is ran which will execute our ApplyVersioningToBindingFiles task. This will modify the binding file correctly to replace the version numbers with the appropriate label from Cruise Control.

Configuration Files

In some of our solutions we include configuration files which may have assembly version numbers in which need to be dynamically versioned. We follow the same approach to make the version number 999.999.999.999 if it needs to be dynamically configured.

In the Cruise Control build a task called ApplyVersionToAdhocFiles is called passing in a list of these adhoc files.

Web service description Files

In some of our solutions where we publish web services from BizTalk we use a couple of MsBuild tasks in our build process which will support regenerating these web services each time the build is ran by using the WebServiceDescription.xml which is produced by the Web Service Generator Wizards. We have MsBuild tasks which support WSE2, SOAP and WCF.

When we use this technique we also need to apply updates to any version numbers which may need to change in the WebServiceDescription .xml. To do this we use the same approach of replacing the versions with 999.999.999.999 for the developer build and then Cruise Control will run the ApplyVersionToWebServiceDescription task to do the dynamic updates.

BAM Tracking Profiles

The BAM tracking profile is an XML file under the hood and we again replace the version number of any of our assembly strong names with 999.999.999.999.

When the Cruise Control build runs it will execute a task which will apply the Cruise Control version label in place of our version number and then when the build deploys BAM artefacts they will be already versioned correctly.

(For details of how we integrate BAM into our MsBuild process please refer to: http://geekswithblogs.net/michaelstephenson/archive/2008/06/15/122870.aspx)

BRE Policy Files

When we use BRE we usually take the approach of managing the policy as an artifact in the solution. We use custom MsBuild tasks to import and export the policy from BRE and store it in the solution as an XML file. If we reference any assemblies in the policy that need to be versioned then we will amend the xml that is exported from BRE and make the version numbers the usual 999.999.999.999. In terms of Cruise Control we treat the BRE exported XML file the same as an adhoc file which requires version updates. We then use the ApplyVersionToAdhocFile task to update this

If you would like to know more about how we implement the BRE features of our build process please refer to the following article: ???

!!!!!! vWhere is BRE post !!!!!

Check local build

Finally when we have done all of the above steps we will run a local developer build to see that the solution is fully versioned, compiles correctly and deploys. We also ensure that the build has ran our BizUnit tests and they are all successful.

The next stage is to get Cruise Control to so some extra stuff.

The Cruise Control Integration Bit

When modifying an existing solution to use dynamic versioning, the requirements for the ccnet.targets file and Cruise Control to implement the version updates is exactly the same as when you setup a brand new solution for dynamic versioning. For details please refer back to that section.

Summary

At this point you can see it is a fair about of work to modify an existing solution to use this versioning technique. I think the earlier you do it in your project life cycle the better. You can use things like Replace All to modify some of the version numbers manually when doing the above tasks but be careful when doing that as you may not want to change all version numbers.

Frequently Asked Questions

In relation to this topic the following questions have occasionally came up.

How do I do side by side versioning in development for a new major release?

The approach described in this article is entirely intended to support development of side by side versions of the same solution. Imagine you have version 1.1.0.0 of your application deployed and we know you want to now work on version 2.0.n.n. You know from your understanding of this topic that you will employ a deployment model which will mean you will have multiple versions of some of your assemblies deployed.

The way you could implement this is to add external Library folder within your solution store the version 1.1.0.0 versions of all of the deployed assemblies. You could also store the appropriate development binding files and any other artefacts to support this update. In your build script you might setup and deploy the existing application (v1.1.0.0) and then modify your build script to apply updates to the application by deploying some or all of your newly compiled code.

In development the newly compiled assemblies always have a very high version number they will always be the latest version and the versioning rules will apply correctly. You will probably also do things like modify binding files to be able to work with two versions of the assemblies. In the binding file remember that the way we do this only our assemblies which are versioned 999.999.999.999 are dynamically changed so if you add details of an older version of the assembly to the binding file we will not touch that instance of a string name as the version number within the string name will not be 999.999.999.999.

When your code is checked in and build on your build server Cruise Control would apply the correct 2.0.n.n (provided you have updated the major number in its config) version number and your build will allow you to have a correctly configured and working side by side versions of the application.

What's the 999.999.999.999 thing all about?

In applying dynamic versioning to the solution we need to be able to ensure that in our development environment a developer can work on a solution which may have side by side versions. We have chosen to use 999.999.999.999 to indicate this is a developer machine only version of the assembly and it is given a high version number to ensure that when it is build it will always be the latest. Although the build tasks accept a regular expression for the version number which needs to be replaced we find it keeps things simpler by always using this fixed number.

Because we use source control we can not apply dynamic versions on a developers machine local build because the files we need to modify will be read only. A developer will always build version 999.999.999.999 of the solution. When the developer checks code in Cruise Control will replace all of these developer 999.999.999.999 versions with the build label from Cruise Control.

BizTalk 2009 Update

Following the release of BizTalk 2009 CTP I have had a brief look at how the new version would affect this versioning approach. In my opinion the approach stays very much the same with the exception that now BizTalk project files are MsBuild based and you can use the AssemblyInfo file to version the assembly rather than the old BizTalk project file. Based on this the change to our implementation would be to not use the old VersionBizTalkProjectFile task and just use the C# one for BizTalk projects as well.

Summary

As you can see from this article this isn't a trivial undertaking. If you do it right from the start of the project then hopefully you will find things a lot easier.

Hopefully this article will plug the gap I identified at the start and there is now a detailed walk through of applying effective dynamic versioning to BizTalk projects which use continuous integration. If you would like to look at the code for the MsBuild tasks we used to help us implement this approach then please refer to the following location:

http://www.box.net/shared/7ebfbdjau0

(Note some companies block access to box.net so if you are unable to get the sample please email me via the blog)

posted @ Thursday, December 18, 2008 4:03 PM | Feedback (1) |

Introducing the Configuration Management Tool

A while back I discussed the approach we use on some projects to configuration management and how we solve the problem of configuring binding and configuration (and any other) files for different environments. I had originally written the linked article on my blog. While it was fairly successful the main project I've been working on has a number of BizTalk projects within it and I felt that the configuration dictionary approach we were using was becoming a bit of a pain as the XML dictionaries were awkward to maintain and you couldn't get a clear view of your configuration.

To help tackle this problem I wanted to create a tool to help with this, the aims of the tool were:

• Provide a nice simple to use UI which would allow me to manage configuration settings for multiple code bases on large projects
• Provide an easy way to integrate a build process into the use of this configuration store
• Try to enhance the previous approach by allowing you to import configuration as the XML files which people may already be using.

The tool is available on codeplex: http://www.codeplex.com/ConfigSettingsTool

Here are a few points to describe the model for the configuration data:

• The system will store information for multiple projects
• Each project can have multiple environments associated within
• Each project will have multiple solutions (code bases) associated with it
• Each solution will have a list of settings which can have default values
• For each solution within a project you can specify setting values for each environment, or inherit the default setting

In the rest of this article I will describe the use of the tool, and how we implemented it for our BizTalk projects. Note that nothing about this tool is BizTalk specific so you could easily use it for .net projects etc. Also in this article I have used screen shots from a real project but have had to censor some information from the screen shot.

Prerequisites

In order to run the tool the prerequisites are as follows:

• The database should be setup on a SQL Server 2005 database (ensure you configure backups for your database)
• The client application requires .net 3.5
• The MsBuild tasks need to run on a machine which has .net 3.5

Getting Started

To set up the application you should take the following steps:

Setting up the database

In the zip file you will download there is a folder containing the database script. You will first need to create a database (which I'm assuming you can do already) and then use the CreateDatabaseObjects.sql script to setup all of the database objects by running it in query analyser.

Setting up the application

In the zip file there is an application folder. In this folder the ConfigurationManagement.Administration.exe.config file contains settings for the application. You will need to edit the connection string to ensure it points to your new database.

Once you have done this and saved the file you can now run the application by double clicking the ConfigurationManagement.Administration.exe file.

Using the Application

Now you are setup if you open the application you can begin setting up your configuration. If you open the Tools à Management menu you will see options for managing the different aspects described above.

Managing Projects

In the picture below you can see the screen for managing your projects. This is pretty simple, you can add/update/delete projects in the grid. Note any changes are not saved until you close the form.

I normally use 1 project per customer or major project of work. In this project you can configure multiple solutions and I normally relate each configuration solution to a visual studio solution.

Managing Environments

Within a project you can also configure multiple environments. As you can see in the below picture I have configured only a couple of environments. Examples of environments you might configure include (System Test, UAT, Production). The environments screen again is fairly simple in that it allows you to add/update and delete environments. Note again changes aren't saved until you close the form.

Managing Solutions

Within the Project you can also define the details of multiple solutions. The solutions form allows you to add/update and delete solutions. The changes are not saved until you close the form. The below picture shows an example of the solutions within this project.

Manage the Solution Dictionary

Once your solution is configured you can now set up its dictionary. The below picture shows an example of this. Within the solution you will create a list of setting tags. These tags are things you will put in files to indicate a special key that you will replace. We typically use the format $(-------) to indicate a key. For each setting you can define a default value which will be used if no specific setting for that environment is configured.

In the below picture if we ran this solution dictionary over a file we would look for each setting and replace it with the environment specific or default value. If a setting key could not be found then we will move onto the next tag.

Again the form follows the usual format for adding/updating and deleting settings and as before the settings are only saved when the form is closed.

Manage Environment Settings

Now we come to managing the settings for each environment. We use the below form which allows you to choose a project and solution. This will then execute a cross tab query to create a grid listing the solutions configuration tags in the first column and then create additional columns for each environment as shown in the picture below.

The first column is read-only, but the other columns allow you to define a specific value for each environment. If you leave a cell empty then the default value is used instead.

This form is different to the others by the nature of what it does. You can not add of remove rows, only edit existing cells. Also when you make a change the details are saved when you finish editing the cell.

At this point you have now configured your solution and are well under way to effectively managing your configuration for your development effort.

Template Files to Configure

As mentioned previously you will typically have a template file which you will run the configuration dictionary over to produce the correctly configured file for a given environment. Some examples of files we typically configure on our BizTalk projects are:

• A template of BizTalk Binding Files
• A template of Web.config and policy cache files for published web services
• A template of BTSNTSVC.exe.config

In the below picture which is a snippet of a BizTalk Binding File Template you can see that I have circled in red the setting tags which are in the $(------) format (note that I have had to censor the tag named). We maintain this template as an artefact within the solution and then during our build process we will run the dictionary over this template file to produce the real correctly configured files for each environment.

Integrating the Build Tasks into Your Script

Along with the Configuration Management application you will also find a folder with a dll containing some MsBuild tasks. There are 3 tasks:

The below picture shows the MsBuild UsingTask statements you would need to use to reference the Configuration Management build tasks. You should also see a ConfigurationManagement.tasks file in the folder containing the MsBuild tasks assembly. You can import this .tasks file into your build script using the Import MsBuild statement.

In the next picture you can see a snippet from our build script. In this case we are configuring two files (a web.config and a policycache file) based on information in the Configuration Management database. You need to tell the task the Project, Solution and Environment which you want information from and also the connection string to your database.

In the bottom part of the above snippet you can see how we use the ProduceSettingsReport task to produce the HTML report I discussed earlier. A sample of this report is below:

Working with the old Dictionary File Based Model

While the new application is very useful and we are able to access configuration directly from the database, some teams may still prefer to store the configuration as a dictionary file like in my original blog post. This has the benefit that you can easily keep the dictionary in a source control system and have a full version history for it. Based on this I wanted to allow teams to make this choice and if they still wanted to use the file based approach then could still take advantage of the tool anyway. This is described below.

Importing Configuration from File

You can begin by importing your dictionary file into the Configuration Management application. The below picture shows how you can choose the Project, Solution and path to the dictionary file. When you import this information any existing setting information is updated, and any new data is added.

Exporting Configuration to File

When you have edited your configuration you may wish to export it again to a file to update the file in your solution. Like with the import you can choose the Solution, Project and path then just export it.

Summary

Hopefully you will find this tool very useful and time saving. There are a couple of limitations and future enhancements you may wish to be aware of if you are considering using this.

1. At present there is no additional security other than if you can access the database. In future releases I plan to add the ability to configure reader and editor roles for each solution/environment/project so this can be locked down more. This will make the tool more usable for bigger projects when you want multiple teams to work on this.

    

2. At present I do not audit any changes made to the configuration. This would be a useful enhancement.

    

3. Any changes made to configuration will not trigger a build in your Continuous Integration environment. This is one reason why a team may wish to combine using the tool with the old file based dictionary approach because when the updated dictionary file is checked in a build would be triggered. If you do scheduled nightly builds this may mitigate this.

posted @ Thursday, December 18, 2008 2:27 PM | Feedback (1) |

First Look at BizTalk 2009 Build and Developer Experience

We had a very useful brown bag session with the BizTalk Product Team this week and went through some of the new developer improvements. Fortunately not much of this was under NDA so I thought id share some of what I learnt and my thoughts on this. Ill just keep this in the simple format below:

BizTalk Project Files are now MsBuild

A new BizTalk project file is basically an extension of a C# project file. It looks very similar to a normal MsBuild based project file see the Picture below.

From this picture you can see lots of familiar stuff. You can see from the following picture from the project file that a BizTalk project file imports the C# MsBuild targets as well as some new BizTalk ones by default.

I believe BizTalk projects have their own project type guid in the project file system which indicates them as a certain type of project file. You can see this above in the MsBuild script where the Project type Guids indicate this project contains C# and BizTalk stuff. I would assume here you can also add the project type guid for MsTest projects and then start putting tests in this project also. In practice you probably wouldn't do this, but it illustrated whats possible with project files now being in MsBuild.

BizTalk Projects Can have non BizTalk Artefacts in them

That's right you can now add other artefacts than BizTalk ones. Some examples of this might be you have a C# class, a text or xml file in your project. Because of what I said above about the C# targets this means if you include a .cs file in your BizTalk project then it will be compiled and made available just like any other normal class. There are a couple of questions and limitations relating to this however:

1. In the current CTP you can not Add New Item and add a C# file etc to the project. This is expected to be in future releases, but for now you will need to add the item to the file system then add it as an existing item.
2. Because the BizTalk project file extends on C# if you add a VB file it will not by default be compiled. If however you add the VB MsBuild targets it should be possible to compile these files also. Essentially its just an MsBuild script so you can do what ever you can do with .net projects.

    

Assembly Info

BizTalk projects now have AssemblyInfo.cs type files like any normal C# project. This makes things simpler for manipulating these kind of files. See the pic below, if you notice there is a new assembly attribute called BizTalkAssemblyAttribute which I guess indicates the assembly is a BizTalk assembly.

No more Deployment/Development

One thing which makes me very happy is that the pointless configuration difference between C# and BizTalk projects is now gone. You can see this below.

It would be nice if the Deploy tick box wasn't checked by default also. This is a useful setting for small demo type projects but in real projects you are unlikely to deploy your BizTalk assemblies through the Visual Studio IDE. I guess now we are based on MsBuild it might be possible to create a new project template where this is not checked (possibly a small community project here).

Compile BizTalk Solutions without Visual Studio

In the installation of BizTalk there is an option to install Project Build components. This means you can use these components to build your solution without Visual Studio. This is what a number of people have been asking for to help them implement the good practice on a Build server of compiling code without needing Visual Studio.

Strong Naming Change

The way BizTalk projects are strong named has changed and is now the same as for C# projects. From the picture below you can see this is set through the project property editor and it sets information in the MsBuild project file. Again another handy improvement in that it is just the same as C#.

Common Project Settings

As you can see from the below picture you can see the BizTalk project also has the same settings which are commonly used in C# projects. You can set the highlighted sections so you can set warnings and constants etc.

One of the more interesting things however though is that you can setup the XML documentation for the project. At present this will only do documentation for things like C# files within the project, but I guess in the future this is something that either Microsoft or the community may look at as a way of incorporating BizTalk Documenter style functionality into the compilation process.

Project Migration

When you migrate your project to BizTalk 2009 there is a migration wizard similar to the one for migrating .net code. You can use this or you can also call DEVENV from the command line with the /UPGRADE switch to migrate your code. You seem to get the normal features of the Visual Studio migration wizards. One limitation here is I believe the project migration does not change the project configurations from the old Development/Deployment modes so you will have to do this manually. And also apparently you can migrate from BizTalk 2006 or BizTalk 2006 R2 to BizTalk 2009.

Unit Tests

As I have mentioned in some other recent posts there is a new Unit Tests property as shown in the picture below. This will make the compilation process generate some additional code to help you unit test various BizTalk artefacts. If you would like some more information about how to use these features please refer to my updated BizTalk Testing Guidance on the following link:

http://geekswithblogs.net/michaelstephenson/archive/2008/12/12/127828.aspx

Improved Build Performance

Apparently when BizTalk projects are compiled now the MsBuild will be able to tell if things have changed and only compile what has changed. Maps and Schemas seem to be complied to a .cs file before being compiled into the assembly a bit like Orchestrations could be if you made that tweak in earlier versions of BizTalk. This means MsBuild is able to detect the changes to these files and only recompile them as needed. This should in theory improve the overall solution compilation time. This could make a big difference on those projects where you end up with lots of assemblies.

This is most of what I remember from the session which was really useful, thanks to the guys who put this together.

posted @ Saturday, December 13, 2008 11:09 PM | Feedback (4) |

BizTalk Testing Guidance - Revisited

I've updated my BizTalk Testing Guidance with some new articles based on what we learned today about BizTalk 2009. Please find the latest index below:

posted @ Friday, December 12, 2008 1:42 AM | Feedback (3) |

BizTalk Testing Series – Testing Pipelines and Pipeline Components in BizTalk 2009

A while back in my series of articles about testing with BizTalk I covered a technique for testing pipelines and pipeline components. The background to the approach is still valid in BizTalk 2009, but the technique is improved. To review the previous article refer to the following link:

Pipeline Components - http://geekswithblogs.net/michaelstephenson/archive/2008/03/30/120852.aspx

Pipelines - http://geekswithblogs.net/michaelstephenson/archive/2008/03/30/120853.aspx

In this post although we could do testing of pipelines and pipelines components using the excellent pipeline component test library. While that component is excellent for testing, I think the new model provides a simpler option and to be honest with Pipeline Component Test Library it sometimes took a little bit of fiddling with setting up the pipeline wrappers to get your test to work. My gut feeling is that the new approach will just be that bit more straightforward.

I will extend on the sample from the previous post about testing schemas. So our solution now looks like the below picture containing a pipeline

1. Configure your BizTalk Project for Tests

Just to reiterate this step from the previous post you need to ensure the enable unit testing property is set for the BizTalk project.

1. Adding references to your test assembly

Again reiterating this step, in your test project you will need to add a couple of references, the main one is the new Microsoft.BizTalk.TestTools assembly which is in the Visual Studio Add Reference list. This will allow you to use these new testing features. You would also need to add references to the pipeline assemblies highlighted in the below picture

1. The test code

In the below code you can see that I will construct an instance of the TestableReceivePipeline which is what my custom receive pipeline will derive from. I can then provide documents and schema information and then execute my pipeline. Unfortunately the documentation of these objects doesn't seem to be complete in the initial CTP so Im guessing there are a few more ways you can configure these objects, but they will give you an easy way to test send and receive pipelines.

1. What about Pipeline Components

In the Pipeline Component Test Library there was the ability to be able to construct a pipeline wrapper in code so you could simulate adding your custom pipeline component to different types of pipeline. I'm not sure that you can do this in the new features of BizTalk 2009, however you can just create a pipeline artefact and add your component to that. The other cool feature is you might not (haven't tried this) even have to add a new project for the BizTalk artefact as you can have BizTalk and C# files in the same project.

Summary

As you can see this new technique appears like it will be a simpler way to encourage testing.

The sample for this demo is available at the following location: http://www.box.net/shared/7rb81bcxsm

If you have problems accessing it then let me know.

posted @ Friday, December 12, 2008 1:33 AM | Feedback (3) |

BizTalk Testing Series – Testing Maps in BizTalk 2009

A while back in my series of articles about testing with BizTalk I covered a technique for testing maps. The background to the approach is still valid in BizTalk 2009, but the technique is improved. To review the previous article refer to the following link:

http://geekswithblogs.net/michaelstephenson/archive/2008/03/30/120851.aspx

In this post although we could do testing of maps, to be honest it was a little bit of a pain. In the newly released BizTalk 2009 CTP there have been some improvements in how you can do some of the testing and I would like to demonstrate this. I will walk through the steps to test your schema.

I will extend on the sample from the previous post about testing schemas. So our solution now looks like the below picture containing an additional schema and a new map.

1. Configure your BizTalk Project for Tests

Just to reiterate this step from the previous post you need to ensure the enable unit testing property is set for the BizTalk project.

1. Adding references to your test assembly

Again reiterating this step, in your test project you will need to add a couple of references, the main one is the new Microsoft.BizTalk.TestTools assembly which is in the Visual Studio Add Reference list. This will allow you to use these new testing features.

1. The test code

In the below picture you can see that I am doing two things. The first is that I create an instance of the new TestableMapBase class which my map derives from. I can use this to execute my map by passing the details of an input file. The results are written to the output file.

I'm also using the schema validation to confirm that the message is valid that comes out. I think there are also properties of the TestableMapBase object which can do this.

In practice when I use this technique I will probably execute the map with a known input file and then compare the results of the output file with a predefined example output file so rather than just validating the schema I'm actually also ensuring the content is exactly as expected. You can do this with a binary comparison of the file.

Summary

As you can see this new technique just simplifies what we had been doing in BizTalk 2006 R2 and is a very nice interface to support this testing.

The sample for this demo is available at the following location: http://www.box.net/shared/yro0poto3m

If you have problems accessing it then let me know.

posted @ Friday, December 12, 2008 12:45 AM | Feedback (0) |