Friday, May 18, 2012
Ive been speaking to a few people starting their organisations first BizTalk projects recently and talking about Host setup and what is a good intial setup and how it may change over time.
Normally you see companies go one of two ways to begin with:
1. They just have one BizTalk host with an instance on each server and use it for everything
2. They have many instances for everything and use way more than they need to.
Im a big believer that a little thought about this and how your company plans to use BizTalk over time can help ensure you dont end up messing around with your host setup constantly.
I was saying to the team that probably the best way to start is create the following hosts to begin with.
- A host called GenericRecevice which would be a host that your in process receive adapters can use
- A host called GenericApplication which is a host for all of your orchestrations
- A host called GenericSend which is a host for all of your send ports
- A host called Tracking which is used for nothing but tracking.
I recommended they left the BizTalkApplication host because it can be the default incase anyone doesnt follow the above rules but do not have any instances of this host on any servers so your team are forced to fix configuration mistakes and to follow your host rules.
When you setup your host instances this obviously varies where you put each host instance based on the topology of your BizTalk group but having already splitt out your artefacts like suggested above makes it easier to move your hosts around in a consistent manner.
At this point you not have a clear well understood host setup and some simple rules around when you would use each one, however the key thing is that there will often be potential exceptions to these rules and you should ensure that if you want to create a new host for one or more artefacts you are able to explain clearly the reason for this new host and not just have a new host because the developer used a different name like Ive seen happen before.
Lastly a thought on the isolated host. We often tend to just have the one BizTalk Server Isolated Host which comes out of the box and run different adapter types on seperate app pools in IIS as an example. This will cover the majority of cases but of course again there will be occasions where you have a need to do something different which is fine so long as its clearly explained.
I would say that generally good reasons for new hosts out side of this patten will be:
- Performance reasons
- Security Isolation
- Adapter specific scenarios
Just a final note that this is intended as some advice when you start your companies first BizTalk projects. If you set of with this configuration it should cover you nicely for most of your cases.
Friday, May 11, 2012
This week I was working with a customer and one of the interesting things in the organisation is that they have a fairly large investment in BizTalk but in general few people in the organisation are knowledgeable about how important a role it plays. I decided it might be interesting to do a small PR exercise; I would write a little post with 10 bits of information about their BizTalk investment that they might not be aware of. It was also interesting that I found out a few things that I didn't know myself in terms of the stats.
The idea of the post was as follows:
- It should be written in a not overly technical language so it could be given to business people
- It should talk about some of the business processes
- We would like to talk about some technical things but it must be explained with examples which are not technical
- It might reference things that the organisation does particularly well
Below is an example of the article for a fictitious company called Acme.
10 things you might not know about BizTalk at Acme
BizTalk is a black box which plays a very important part in the Acme application architecture. A lot of people have probably heard of it but don't know much about what it does. This article will help to tell you a little more about what it is and its role.
1.
What is BizTalk
BizTalk is a product from Microsoft which is a centralised Integration Server platform. It is used to develop solutions to integrate your applications.
As a platform it allows you to develop solutions from simple application connectivity to complex ESB, hybrid Cloud or B2B scenarios. The platform allows you to host these integration solutions side by side so they can benefit from the investment made in each other.
BizTalk provides many "quality of service" features such as a scalable hosting environment and administration and management tools which let you support solutions in production.
2.
Where does it fit at Acme
At Acme over the years we have developed many integration solutions with BizTalk. Some of the bigger better known solutions include "process a" instances between application A and application B most of the interaction between these applications happens via BizTalk messaging and workflow. BizTalk deals with complex message transformation between the applications, provides durable messaging to handle any server downtime and fault handling in addition to workflow to manage the system to system integration processing.
In Acme almost all documents which go to customers will go via an application called "application c". BizTalk enables the integration from "application a" into "application c" and BizTalk handles both real time document requests and heavy duty batch processing. BizTalk makes it easier for these two applications to work together to deliver the end customer value.
In Acme approximately 5000+ "process a" instances are processed per day from B2B partners and loaded into "application a".
In Acme all payments from customers go through BizTalk in both real time and for monthly payments.
All orders Acme receives are processed through BizTalk
BizTalk is also involved in many other processes.
3.
Number of messages processed
It is estimated that BizTalk has processed in the region of n messages since it was introduced to Acme.
[Note: this was an estimate based on the average number of messages per day from HAT queries]
4.
More Stats
Since BizTalk went live it has processed approximately:
- £n worth of customer payments
- n documents have been sent to customers through BizTalk
- n orders from B2B partners have been processed through BizTalk
5.
Long running transactions
BizTalk excels in the area of long running transactions. At Acme we have some processes which can take hours or even days to complete. Because BizTalk is able to maintain the state of a workflow the long running process can do some work and then stay in a dormant state until the next part is due to execute or until an application sends the response message back to trigger the next part of the process.
This state persistence is also able to withstand failure conditions or even the whole data centre going down and a disaster recover scenario happening.
6.
Load Levelling
One of the key features of BizTalk which plays a very important role in the application performance at Acme is a pattern called load levelling. If you imagine that in some areas of the systems you have different amounts of activity at different times. For example overnight from 10pm onwards we get a large spike in the number of "process d" and "process e" instances to support the overnight batch processing. During the day these processes also execute but it is a much smaller volume. Overnight the input into BizTalk is at too high a rate to be passed through to "application e" at the same rate without significant scaling of these applications. In BizTalk we flatten this load and deliver it to the end system at a controlled rate which the application can manage. Imagine pouring water through a funnel.
7.
Fault Handling
Typically applications are prone to errors and downtime. One of the most important features of BizTalk is its ability to recover from errors in an application it is trying to integrate with. If for example "application a" threw an error when BizTalk tried to send a message to it then BizTalk can work out what kind of error it is and if an automatic retry can be attempted. The error may persist and indicate a bigger problem. If the number of automatic retries is exhausted then BizTalk will suspend the message but provide a management interface so that the support team can see the problem, workout what went wrong and can start the retry process again when the issue is resolved.
8..9..10….
Ive removed the last few as they were a little too specific to the customer to display on my blog, but hopefully you get the point.
This communication actually went down really well and I think served as an excellent way to tell people a little about what this expensive black box they occasionally hear about is and why it's important to the organisation.
Has anyone else done anything similar?
Friday, May 04, 2012
Name: | Don't automatically shove your config in BTSNTSVC.exe.config because its easy? |
Description: | I wrote a blog post a few years ago around the options for where you could put configuration settings in BizTalk (Click here). As I mention in the blog post its very common that people just fill the BizTalk config file with lots of settings because its the easiest option. I often work on big projects where you have many BizTalk projects running at the same time with different delivery deadlines. One of the problems is that the BizTalk configuration file is one of the few resources that is shared across your applications no matter what. When you have settings that are in the BizTalk config file then your BizTalk application now has additional dependancies so if you want to produce a new version of the file you have to be careful to consider how changes to the file may affect other BizTalk applications. |
Symptoms: | - You have a big project with many BizTalk projects running at the same time
- You have large development teams
- You have different applications running on the same group
- You have poor dependancy management between your projects
- You dont have much governance around changes to the BizTalk config file
|
Pain: | - A change for one application may break anothers functionality
|
Cure: | - Dont take the quick and easy option just because its quick and easy
|
Im not saying its bad to use the BizTalk config file, far from it. What I do think is a good idea though is to have a development process and governance around changes to the BizTalk configuration file to ensure a quick and easy change doesnt bite you in the future. If you want to put something in the file for custom application configuration make sure you talk it through with your technical lead or others on the team to ensure everyone agrees that the BizTalk configuration file is the right place to put it. Also have some guidelines and standards for your team to determine what types of configuration should go where.
Tuesday, April 24, 2012
|
Name: |
Don't over complicate your map
|
|
Description: |
I was recently looking at a BizTalk map which had a bug in it and one of the things that I notices was that the developer had strung together a number of functoids to do some logic in the map. We were talking about 12 functoids being used to workout which value should be in the output field.
I think sometimes developers are a little frightened to drop into .net code feeling that it is perhaps not the BizTalk way. In this case the logic could have been executed in a .net assembly with about 4 lines of code and called via the scripting functoid.
I'm not saying that you should not use the out of the box functoids but I think there is a balance where making a very complicated stringing together of functoids should be a trigger to yourself to question if there could be a simpler way to achieve what your trying to do. |
|
Symptoms: |
|
|
Pain: |
|
|
Cure: |
- The first thing to do is ensure that you recognize that it is complicated. If this is the case I would recommend asking a colleague to review your map and to see if they find it easy to understand. If they don't then you have some work to do
- By putting the logic in our case in a .net class we could easily test it with some unit tests to ensure the logic is rock solid before it gets anywhere near the map
|
Just as a point to note on this post, there is also the excellent and similar post about the Everything but the kitchen sink anti pattern used in maps. The difference between my post and that post is that I am saying just use the scripting functiod if you need to so that your map doesnt become overly complicated. The other post is saying dont put things in a map if they dont belong in a map.
That post is available on the following link:
http://blogs.msdn.com/b/ebattalio/archive/2006/11/16/anti-pattern-kitchen-sink-maps.aspx
Sunday, February 26, 2012
I've been reflecting on some of the occasions over the years where I have had to step into projects and get them back on track. I've had these kind of situations in different fashions such as a customer I work with has asked me to move to another project because they were having problems or a new client who has contacted me specifically because they have ran into difficulties on their project and need some help.
I tend to find that these kind of projects fall into two types. The first is where the team cant see the wood for the trees and have got so deep into their problems that they just need an independent person to come in with a fresh view and ask some common sense questions or some special technical expertise which the team doesn't have. The second type tends to be where the project has stalled and is not making progress and things get very political, in these cases the project tends to need someone to come in and to challenge the way things have been done and make some changes to the working practices so things can be more successful.
I've been thinking about some of the situations where I have engaged on these type of projects and what some of the common symptoms are. Below is a list of the most common ones I have come across and I'm wondering what other people tend to find.
Thursday, February 23, 2012
I have been wanting to talk for a while about the approach we take to analysis of an application we are about to integrate with. In the past I have seen a lot of cases where analysis is undertaken by looking at the interface specification documents from an application vendor and then taking their contract (which hopefully is standards based) reading them numerous times and then we plough into the development. The big problem with this is that we make a number of assumptions about the application and its interface which could significantly affect our development if these assumptions prove to be incorrect or we have missed something.
One of the approaches I like to take is to understand the contract but to also consider the behaviour of the interface as much as the contract. I want to also test my assumptions and interpretation of how this interface will work. To do this I like to use SpecFlow which you may have seen me blog about on a number of occasions. As part of the analysis phase I would develop some tests using SpecFlow and visual studio which will describe the behaviour and assumptions we are making about how the application will work. You could imagine a scenario like the following:
Feature: Load a Journal Message into Peoplesoft
So that | I can Participate in the journal integration process |
As a | BizTalk Application |
I want to | Load a journal message into Peoplesoft |
Scenario: Normal Case
Given | Peoplesoft is in a normal state |
And | I have a standard journals message |
When | I call the Peoplesoft API Journal Load |
Then | Peoplesoft will accept the message |
And | Peoplesoft will return an IBResponse message |
And | The IBResponse message will contain a success status code |
And | The journal will be in a processing state in Peoplesoft |
Scenario: Invalid Journal Type
Given | Peoplesoft is in a normal state |
And | I have a journals message which has an invalid journal type element |
When | I call the Peoplesoft API Journal Load |
Then | Peoplesoft will return an IBResponse message |
And | The IBResponse message will indicate a status of FatalError |
And | The journals message will be in a failed state in Peoplesoft |
The above gives us a great yet simple and effective description of how we use the API.
A lot of the vendor API's I have come across tend not to go to this level of detail or it would often be buried in a huge documentation pack. What we want to do is extract this information and our assumptions about the behaviour of an API and how it relates to our requirements and present it in a simpler way. In some cases a feature we require may use two methods on an API so the above technique would be a good way to document this relationship.
Once we have defined the behaviour we expect from the API using the gherkin syntax and Specflow in Visual Studio we can then implement code behind each of the Gherkin statements which will implement this step. Suddenly we have some acceptance tests which will validate our assumptions that an application API behaves the way we expect it to. By doing this early in the analysis phase of your project you can save yourself a lot of problems later on by reducing the risk that your assumptions are incorrect.
The output of this approach is that we will have the following:
- Lightweight documentation of how we will use the API and what we expect it to do
Tests which prove the API does what we want and can be used for regression testing
Real world Example
Now on to a real world example where this approach proved to be very valuable. At one customer I have worked with there was an application where we had used BizTalk to integrate with it. The customer also had 5 other legacy applications which integrated with it directly in a point to point fashion and not via a broker so they were tightly coupled to the application. There was a project to upgrade this application which the vendor had told us would be fully backwardly compatible and that none of our applications would be affected but the vendor did supply a new version of their interface specification which was a 50+ page document. All of the legacy application teams impact assessed the upgrade based on the interface specification and said that they would not be affected. We took a different approach. I told the project team that we would run our behaviour tests against the new version of the application which had been installed for a proof of concept. This would tell us if the application was backwards compatible. Based on the results of the tests we would then know how much detail we needed to go into around the interface specification and how risky we would view the upgrade to be. When we ran the tests surprisingly half of them failed with significant failures.
At this point we engaged with the vendor and begun to go through the test failures. We quickly found out the problem which the vendor replied "Oh you do it like that". It turned out that the vendor had architected part of their product and a feature which the BizTalk and all legacy applications mentioned above used had been removed from the product. The issue was dealt with from there and the solution isn't really relevant to the context of this post, but hopefully you can see the key point that without this approach the customer would have gone a significant way into this upgrade project and spent plenty of money before realising that they had a major compatibility problem whereas within 10 minutes of starting our impact assessment we identified this problem because the tests failed.
Needless to say the vendors backwards compatible statement wasn't accurate and their interface specification had been poorly maintained.
This approach can be can be used for simple or complex stuff and doesn't require a lot of investment but can save you a lot of time in the future.
Wednesday, February 22, 2012
A few days ago I read an article by Richard Seroter comparing the different cloud storage options and comparing Windows Azure and Amazon S3. I commented on his blog about how Id like to see companies use these more and more for B2B data exchange when you have a batch file rather than the traditional solutions using FTP and the painful infrastructure piece that often goes with this kind of project. The normal challenges include:
- Who hosts the FTP service or do we both
- What kind of security do we use
- Are their any special infrastructure requirements eg leased line
FTP based solutions are often a pain unless the client has a well setup and established service and then there is the assumption that your B2B partner also has a similar capability in place.
From the comment on Richards blog post and some thoughts about requirements we may have at one of my clients I began thinking about this as an option to seriously consider. I also wondered what there was available in the Amazon space as it's a couple of months since I have looked at the Amazon cloud stack.
Following a quick search I came across the NSoftware adapters which had a ready made BizTalk adapter for Amazon S3
http://www.nsoftware.com/products/biztalk/adapters/s3.aspx
From here I thought it would be nice to have a walk through setting up a simple B2B scenario involving Amazon S3 and BizTalk.
Walk-Through
In this scenario we will have a business partner (Contoso) who will send us a file containing a batch of orders. The business partner will upload the file to a bucket in AWS S3 with the file name ending with a .input extension.
BizTalk will poll the bucket with a receive location configured to look for files with the .input extension and then pull the file down into BizTalk on premise in our company (Acme). In our BizTalk implementation we will simply push out the file to a folder to simulate the orders being sent to the order processing application. When the order processing application has finished processing orders it will output a response file which BizTalk will collect and push back out to the bucket in AWS S3 with a .output extension. Contoso will then be able to download this file as required and process it in their applications.
Setting up Amazon S3
I'm not going to go too much into the configuration and setup of AWS S3 but it didn't take long at all to set this up. Once you have an AWS account you can subscribe for the S3 service and setup a bucket. When setting up a bucket you need to choose its location. A bucket is simply like a root folder where you will put files and you can create files with a folder type key structure which will make them appear to be in folders within the bucket.
At time of writing AWS has a pretty good deal on the S3 package, particularly if you are a low user where you will be paying very little if anything if you are only playing around with the service or are only transferring/storing up to a certain amount of data.
(Make sure you review the pricing page for this though so you know where you stand).
Installing the NSoftware Adapter
The S3 adapter is available from NSoftware under a commercial and community license in their adapter bundle. For this post I was experimenting with their community edition. The install of the adapter was very straightforward and also includes the setup of a few other adapters which come together with the bundle.
I was using the adapter pack on BizTalk 2010 running on Windows 2008 R2.
Setting up the BizTalk Process to Receive from S3
The receive process was very simple to setup I firstly created a receive port which with a receive location which used the S3 adapter. The configuration settings I used are as follows:
Setting | Description |
Access Key | This setting is the access key for your S3 account which you wish to access the bucket with
|
Secret Key | This is the secret key from your S3 account which you wish to access the bucket with
|
Bucket Name | This is the AWS S3 bucket which you want to access. The bucket is like a folder in the cloud which you want to access.
|
Object Mask | The object mask indicates the path type information for files you want to look for. This setting caused me some confusion initially because I wanted to have folders within my bucket initially and I had been trying to specify the folder as part of the bucket name property. I have provided some more samples combinations below to help explain this setting.
|
Delete Mode | This lets you specify what happens to the file in the cloud when BizTalk has collected it. If you don't set this to something other than the default then BizTalk would continue to download the same file over and over so I am not sure why you would want this setting to be anything other than Delete on Success.
|
| | |
To provide some more information on the object mask property here are some examples:
Scenario | Example |
Collect all files in the bucket
| * |
Collect all files of the .input extension in the bucket | *.input |
Collect all files of in a folder called "in" | in/* |
On the send port side I had a port configured with a subscription for all files from the Receive port we have setup above and the send port would simply send the inbound file to a folder on the BizTalk developer machine. We will call this folder c:\LOBApplicationIn
Simulating the LOB Application processing the file
Once the BizTalk send port has written the file to the local disk simulating delivering if to an on premise line of business application we will simulate an application processing the file and making a response available for BizTalk by simply copy and pasting the file to a different folder. We will call this folder c:\LOBApplicationOut
Setting up the BizTalk process to send the file back to S3
Once the output file from the application is available we will setup a BizTalk receive port and location using the File adapter to poll for files coming out of the LOB application. These files will be picked up and sent to a send port which will use the S3 adapter to send the file back to S3.
In setting up the send port I simply used a subscription based on the send port name, however of more interest are the configuration properties of the S3 adapter which are shown in the below table.
Setting | Description |
Access Key | This setting is the access key for your S3 account which you wish to access the bucket with
|
Secret Key | This is the secret key from your S3 account which you wish to access the bucket with
|
Bucket Name | This is the AWS S3 bucket which you want to access. The bucket is like a folder in the cloud which you want to access.
|
Object Key | This is the name/key of the file you wish to create. You can configure this with folder type semantics if you want to create folders in S3 (remembering that this is just a representation of the key heirachy).
|
One point of interest here is that the object key property supports most of the macros that the file adapter supports so you could set the object key to be something like %MessageID%.xml to create a unique file name each time for an xml file.
Business Partner uploads a file
The final part of the walk through is kicking off the process. I could use the AWS SDK to create a client and upload files but just as easily I can use the AWS console to manually upload the file. The good thing about this is it shows that in a B2B scenario this would support low-tech business partners who may simply have a user who uploads and downloads files as well as someone advanced with system integration to publish and read from S3.
When the business partner uploads their input file they could monitor the bucket and once BizTalk had pushed a response file back to S3 the partner would be able to download it as required.
Considerations around AWS S3
Following the walk through I thought I would make a few observations and comments about things I would consider when using adapter in a real world scenario.
High Availability for the BizTalk download
One of the points to consider when using this in a production like setup is configuring the receive ports for high availability. The way the S3 adapter works is very similar to the BizTalk FTP adapter in this aspect. You would want to setup a clustered BizTalk host to run the receive location on so that you can achieve high availability but not have to worry about the same file being picked up twice if it was concurrently read from two different servers.
What about multiple partners
If you were working on a scenario involving multiple partners I think it would be good practice to keep different partners data in separate buckets in S3. Looking at the way the security permissions work for S3 this would seem to be a sensible way to ensure that partners can not access each others data. (Happy for someone who may know better to challenge this assumption).
Bucket Permissions and ACL
I think before doing a real project involving this adapter you should have a clear understanding of the ACL usage for buckets. I would refer you to the following link for more information:
http://docs.amazonwebservices.com/AmazonS3/latest/dev/UsingAuthAccess.html
Summary Notes
In a proof of concept scenario I found this adapter really easy to work with and quite nice. To a BizTalk developer or administrator it looks very similar to the File and FTP adapters which is a good thing.
I was able to get my scenario up and running very quickly.
Useful Resources
The following links are some of the resources which may be useful to you as follow ups to this article:
AWS S3 Bucket documentation:
http://docs.amazonwebservices.com/AmazonS3/latest/dev/BucketAccess.html
AWS S3 API:
http://awsdocs.s3.amazonaws.com/S3/latest/s3-api.pdf
S3 Adapter:
http://www.nsoftware.com/products/biztalk/adapters/s3.aspx
S3 Adapter Community Edition:
http://www.nsoftware.com/products/biztalk/community.aspx
Saturday, February 11, 2012
Ive recently really seen some of the benefits in action from the work we have been doing to introduce Behaviour Driven Development techniques into our BizTalk development process.
A couple of months ago one of the members of our team was unavailable and he was the expert with one of the BizTalk Applications we had developed. The team was small and we have a lot of processes and this particular application was quite complex and no one had the level of experience that this team member had with the business process in this particular application. After he was unavailable we got some new requirements for this application and it was a daunting prospect to pick up the reins and try to make these changes without this valuable member of the team around. I know that in an ideal world you would have a number of people with lots of experience on each project but in the real world we had a small team, lots of projects and lots of change so its tough to be in a great position. What we did have was some good documentation and lots of BizUnit tests but it was still a very challenging change.
To try to address this problem one of the first things I did was to allocate some time so that we could refactor all of the BizUnit tests to use the the BDD approaches I have been encouraging and also the "testing whats happening inside of BizTalk" testing approach. The idea here was this small amount of investment would help us to understand the BizTalk application and its implementation of the various processes but also put a tight level of Gherkin style documentation around the BizUnit tests.
After about a week of refactoring this (and it was a challenging week) and implementing some of the new change we needed to deliver I handed over some of the work on the new change to my colleague on our team. I was very impressed at how much simpler the hand over was and how well Francis was able to get his head around how the application worked.
That week of refactoring really moved us from a position where I had been reluctant to touch anything in that particular BizTalk application to a position where I am very comfortable we have a good handle on it now. The BDD initiative we did in 2011 was definately one of the best changes we have made to our development process.
Some links to some of the techniques we have used:
BDD with BizTalk 2010
Part 1 - http://tinyurl.com/84j78mo
Part 2 - http://tinyurl.com/6n72toy
BDD with BizTalk 2006
http://geekswithblogs.net/michaelstephenson/archive/2011/11/04/147577.aspx
Testing Whats Happening inside of BizTalk
http://tinyurl.com/82wcskm
Friday, February 03, 2012
Before I get into this I am not an Office Live Small Business or Office 365 expert but I have used Office Live Small Business for number of years as a light weight way of managing some parts of my business. I now need to migrate to Office 365 and one of the areas which concerned me was around custom lists which had attachments. I had a number of these with lots of rows and I had been waiting for some information on how these would be migrated and hoped they would just be migrated for me automatically.
When the migration guidance came out this was unfortunately not the case so I thought this post may help some others who have the same task to perform over the coming months before the final closure of OLSB.
Step 1: Replicate the list structure
In your Office Live Small Business account open up the custom list and go to its settings so you can see the structure of the list.
In your Office 365 account create the new custom list to replicate the structure from Office Live Small Business.
A couple of recommendations here:
- You may want to consider leaving the constraints on columns such as choices etc for later and initially create the columns with simple types then fix them later
- Ensure the columns are in the same order to make things simpler
Step 2: Open Your existing list in Microsoft Access
In your Office Live Small Business custom list use the Actions menu and select Open with Access.
This will open the list in Microsoft Access so you will obviously need it installed.
When the list opens in access choose the option to export a copy of the data and an appropriate location to save it.
Step 3: Open your new list in Microsoft Access
In your Office 365 account open your list and select the Open with Access option highlighted below.
When the list opens choose to have data linked to the sharepoint site like in the below pic.
Step 4: Copy the core data
Unfortunately you cannot copy everything all in one go because the rows must be created before an item can be attached. From here the next thing to do is to select the column headers in the access instance from the Office Live Small Business list. Select all headers except the one for the attachments column. Like in the below picture.
Next go to the Access instance which is linked to the Sharepoint List in the Office 365 site. Paste the lines into this Access instance. The paste action will slowly insert the new rows into the Office 365 SharePoint List.
Step 5: Copy Attachments
You should now have two Microsoft Access databases with the same rows in. One a copy from Office Live Small Business and one with a linked table to the Office 365 SharePoint list. The rows will all be in the same order.
The next step is to copy the attachments.
In the Office Live Small Business Access instance select the column header for the attachments. The one with the paper clip symbol as its header. Then copy the entire column.
In the Office 365 Access instance select the attachments column header and paste the entire column.
This paste action will probably take a while if you have a lot of attachments.
Conclusion
As I mentioned at the start the biggest concern I had with the Office Live Small Business to Office 365 migration was around what to do with all of my custom lists and attachments. Unfortunately the self migration guide wasn't very useful for this bit because It just said to export your data but didn't cover importing it or anything about attachments.
As you can see this only took a short time to do so generally I am quite pleased that this is complete and I can now get on with enjoying the many new features of Office 365 and let my OLSB account disappear into the past.
Sunday, January 15, 2012
I've been working on a little project recently and one of the requirements was to be able to identify someone's influence in the social media space. I happened to come across Klout which if you are not familiar is a service where people can connect all of their social networking accounts together and it will then analyze your public activity and workout how much Klout you have. Even if you have not joined Klout it still looks at public Twitter information and can rate users just based on their public Twitter activity.
This sounded like a service worth investigating because it looks at who you follow and who follows you and things like retweets and mentions. It then rates you. This sounds like a good service based way of finding out how influential a user is.
The aim of this article is as much for myself as a reminder of this simple POC just to retrieve a score.
Start off by registering your application by signing up with the Klout developer site.
http://developer.klout.com/
Next the below class can wrap up interaction with Klout. You can see in the get user ratings method I take a list of user names and then format the url and make calls to Klout. The response is loaded into a Linq to XML XDocument so I can easily extract the score. A dictionary is returned of the user ratings.
There is probably tidier ways to do this and im sure we will refactor the code etc as we go through the project but for now the little sample below may be useful to someone as I haven't seen many C# samples online.
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Net;
using System.Xml.Linq;
using System.Xml;
using System.Diagnostics;
namespace Acme.Klout
{
///
<summary>
/// Wrapper for the klout service
///
</summary>
public
class
KloutClient
{
private
string KloutKey
{
get { return
"[ADD YOUR KEY HERE]"; }
}
public
Dictionary<string, decimal> GetUserRatings(List<string> users)
{
const
string urlFormat = @"http://api.klout.com/1/klout.xml?users={1}&key={0}";
var response = new
Dictionary<string, decimal>();
var scoreDecimal = decimal.Zero;
foreach(var user in users)
{
var url = string.Format(urlFormat, KloutKey, user);
try
{
scoreDecimal = decimal.Zero;
var kloutResponse = GetKloutResponse(url);
var document = XDocument.Load(kloutResponse);
var score = document.Descendants("kscore").First().Value;
scoreDecimal = decimal.Parse(score);
}
catch (Exception ex)
{
Trace.WriteLine("Error getting klout rating: " + ex.ToString());
}
response.Add(user, scoreDecimal);
}
return response;
}
private
Stream GetKloutResponse(string url)
{
var request = WebRequest.Create(url);
var response = request.GetResponse();
return response.GetResponseStream();
}
}
}
Saturday, December 17, 2011
Ive been doing a little stuff with the Twitterize library and the twitter API.
I started unexpectedly getting a 401 unauthorized error when things had been working fine previously.
Eventually I tracked this down to be an issue with clock sync. I am using VM Ware fusion on Mac and when if suspended then resumed my VM the clock wasnt always in sync. When this was fixed it all works fine.
Note to self the way to change the VM is
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021718
Saturday, December 10, 2011
For a while I’ve been working on an idea and whitepaper around analyzing source code to measure the size and complexity. There are some tools around which can do this but my differentiator is that I was interested in BizTalk projects and wanted to look at how these could be measured. Note however this can be easily used for non-BizTalk projects too.
The aim is to be able to understand the size and complexity of a solution and to allow some real metrics to contribute to your understanding of the Total Cost of Ownership of your solution. Using the tools I have developed alongside this I have been working with some customers to integrate the analysis into the build and continuous integration processes so that we can produce a report of the code and also track changes over time to monitor how the code based vary in terms of size and complexity.
The white paper goes into the detail of the theory about how we measure code and is available on the following link:
https://skydrive.live.com/redir.aspx?cid=983a58358c675769&resid=983A58358C675769!1820&parid=983A58358C675769!1819
The source code and application/tools which can do this for you are available on the following codeplex project site
http://biztalksolutionanaly.codeplex.com/
At this stage the project is being used to monitor approximately 20 code bases at two companies who have significant investments in BizTalk and I’m also looking for feedback from the community over other things we should measure.
It would be great to hear how you get on with this or if you have any views on it.
Thursday, December 01, 2011
A while back I did the user group sessions and cloud case video about Behaviour Driven BizTalk Development and talked about how you could use SpecFlow with Visual Studio 2010, BizUnit 4 and BizTalk 2010 to create acceptance tests for your solution.
The video is on the below link:
Part 1: http://cloudcasts.cloudapp.net/ViewWebcast.aspx?webcastid=2520897495056204879
Part 2: http://cloudcasts.cloudapp.net/ViewWebcast.aspx?webcastid=2520897493960073864
Of the many advantages of this approach one of the best ones was the benefits of using acceptance tests as documentation of the solution. Well written acceptance tests can become excellent, accurate and tested documentation which sits with the code.
So while all is good in BizTalk 2010 world, I've recently been working with one of my BizTalk 2006 customers and wanted to explore how we could do something to bring these advantages to BizTalk 2006. The key limitation for BizTalk 2006 is that Specflow did not support Visual Studio 2005. A few weeks ago I talked in a separate blog article about how we could use xml comments in the BizUnit 2 xml files to be able to annotate our existing tests so that we have a structure which gives a good idea of the intention of a test to sit alongside the implementation.
http://geekswithblogs.net/michaelstephenson/archive/2011/11/04/147577.aspx
So while we are now in a better place I wanted to convert the comments from these xml files into some kind of documentation which I can easily distribute to none BizTalk developers. To solve this problem I have developed an MsBuild task which you can point to directories which contain your BizUnit tests and then it will recursively look for files in those directory which contain BizUnit tests.
When a test is identified we simply use some regular expressions to pull out the feature name, scenario name and gherkin statements I talked about previously and then produce html reports documenting each feature.
Here is an example of a documented test:
Here is an example of the usage of the msbuild task:
<!-- Gherkin Documentation of BizUnit -->
<AppFx.Build.Tasks.BizUnit.DocumentAcceptanceTests
FolderPaths="@(BizUnitFolderToDocument)"
OutPutFolder="$(PackageDirectory)\Documentation\Features"/>
You will need to do the normal importing of an MsBuild task and also setup the properties and item collection for the folders to inspect for tests.
Here is an example of the produced html report:
A couple of points to note:
- If you have some BizUnit tests which do not have the comments in then they are marked in a file canned .html which contains the undocumented tests
- We plugged the task into our MsBuild process so that everytime we produce a build of code the documentation is also included with it.
- Each html file will contain all of the documentation for scenarios which were marked with the same feature name.
Hopefully this will help lower the cost of ownership of your existing solutions and help when people join and leave your development team to help make your BizUnit tests easier to understand.
Download the bits
The code and bits for this article are available in my skydrive samples folder on the following link:
https://skydrive.live.com/?cid=983A58358C675769&id=983A58358C675769%211812
Friday, November 04, 2011
One of our .net application teams has had a problem for quite a while that related to impersonation and kerberos multi-hop delegation which had proven quite difficult to resolve. We eventually resolved this and I thought it would be worth popping a little bit of information about it out there incase anyone else has similar problems.
We had two web services with 2 methods which participate in a Kerberos multi-hop delegation scenario using WSE 2. One of the methods works fine all of the time and the other method intermittently was having problems.
Symptoms
The symptoms we experienced were:
- For method 1 everything worked fine all of the time
- Everything appeared to be working fine for method 2 but after a period of time it would start getting errors which indicated the Kerberos token had expired
- Method 2 also looked occasionally like the wrong user identity had been flown over the wire to the back end service
- The calls to method 2 which stopped working after a period of time seemed to coincide with the expiry of the Kerberos ticket (around a week to 10 days can't remember the exact time frame but it coincided with the cache duration of a Kerberos ticket)
- After an IIS reset things worked fine straight away for a period of time and then would stop working after the time period above
- The error message we were getting was: The Kerberos credential handle could not be acquired. The AcquireCredentialsHandle call returned the following error code: A specified logon session does not exist. It may already have been terminated.
Troubleshooting
This was very painful to troubleshoot, the .net team were struggling to recreate this problem in any of their test environments. Through the addition of some instrumentation they were able to derive what was going on.
What was happening
When the development team originally coded web service A for method 2 they had implemented a singleton or caching pattern of the proxy object for web service 2. This meant they did not need to construct a new instance each time.
This meant that when the method 2 was called then the WSE policy was applied and the Kerberos token used etc but this object must internally be handling things like WS-Conversation and some kind of caching so that the Kerberos token can be reused over the lifetime of that proxy instance. In essence the first user to create the object's identity was used for every subsequent usage of this object when applying a Kerberos token to the web service calls made with this proxy.
Also this would explain why after a period of time the calls fail because the Kerberos ticket obtained for that user would have expired.
Solution
The solution was to change the code in web service A so that the proxy to web service B was not cached or a singleton and so that a new instance of this proxy was created each time.
Would this apply to WCF
At this stage I have not tested if WCF would get the same problem but I do know you run into issues in various configurations if you haven't disposed of a proxy class correctly so generally it has become a very common practice for developers to safely dispose of their WCF proxies so this tends to be less of an issue. In WSE 2 or 3 developers are not used to having to use a dispose pattern around web service proxies
Lessons to learn
Some take away's for this include:
- It is a lot more likely to be your own code than framework code causing obscure problems like this
- Impersonation can be a dangerous thing so ensure you test and code review any usage of it
Credit goes to Matt Buckley who found the needle in the stack of needles in the custom .net application and found the cached proxy instance through extensive analysis of the custom code.
I've been back working with BizTalk 2006 R2 for a customer recently and I've become such a fan of the BDD style acceptance tests I've done in the past with BizTalk 2010 that its quite frustrating working back in Visual Studio 2005 and not being able to use Specflow alongside BizUnit 4 like I described in the recent videos on these subjects
In BizTalk 2006 development your back to the older style xml bizunit tests and we were looking at some old tests written a long time ago and frankly it's hard to remember what they were supposed to do. This is where the separation between intention and implementation of the test is so important. Since there isn't a great deal we can do about this in terms of upgrading the technology we took the very simple approach of annotating the BizUnit tests using the Gherkin style so that before each test step or group of test steps there was something to explain what was intended to happen. The other benefit is it really encourages the TDD approach.
In the first example below you can see when writing a new test we have simply used the Gherkin syntax Given/When/Then to record the requirements and definition of what needs to happen in this test.
<TestCase testName="VAU.CardChanges">
<TestSetup>
<!-- GIVEN: The test event queue is clean-->
<!-- GIVEN: The test folder is empty - Acquirer folders -->
<!-- GIVEN: The test folder is empty - CardSystem folders -->
<!-- GIVEN: The test folder is empty - CRM folders -->
<!-- GIVEN: The acquirer file generation numbers are zero -->
</TestSetup>
<TestExecution>
<!-- WHEN: TWS calls BizTalk to trigger the validation of payments -->
<!-- THEN: The response will indicate that validation has started -->
<!-- THEN: BizTalk will start the batch processing orchestration -->
<!-- THEN: BizTalk will send the batch to the acquirer -->
<!-- THEN: The acquirer will find the batch in their input folder -->
<!-- THEN: The acquirer will put the response in their output folder -->
<!-- THEN: BizTalk will recieve an acknowledgement from the acquirer -->
<!-- THEN: BizTalk will recieve a response file from the acquirer -->
<!-- THEN: BizTalk will process any card updates -->
<!-- THEN: The card will be updated in CARDSYSTEM-->
<!-- THEN: The validation batch will be updated in CARDSYSTEM-->
<!-- THEN: The card will be updated in CRM-->
<!-- THEN: Biztalk will have completed processing the validation batch response -->
<!-- THEN: The card update received by CRM will be valid -->
<!-- THEN: The card update received by CARDSYSTEM will be valid -->
</TestExecution>
<TestCleanup>
<!—Clean Test Folders -->
</TestCleanup>
</TestCase>
The fully completed and implemented test looks like the below example. Now hopefully you can clearly see how important this simple documentation around the tests are for people who may look at this in the future.
<TestCase testName="VAU.CardChanges">
<TestSetup>
<!-- GIVEN: The test event queue is clean-->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.CleanEvents">
</TestStep>
<!-- GIVEN: The test folder is empty - Acquirer folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerResponse</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<!-- GIVEN: The test folder is empty - CardSystem folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CardSystemWS</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<!-- GIVEN: The test folder is empty - CRM folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CRMWS</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<!-- GIVEN: The acquirer file generation numbers are zero -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.DBExecuteNonQueryStep">
<DelayBeforeExecution>1</DelayBeforeExecution>
<ConnectionString>Persist Security Info=False;Integrated Security=SSPI;Server=localhost;Database=PaymentCardServices</ConnectionString>
<SQLQuery>
<RawSQLQuery>UPDATE FileNumber SET FileNumber = 1 WHERE ID = 'NextFGN'</RawSQLQuery>
</SQLQuery>
</TestStep>
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.DBExecuteNonQueryStep">
<DelayBeforeExecution>1</DelayBeforeExecution>
<ConnectionString>Persist Security Info=False;Integrated Security=SSPI;Server=localhost;Database=PaymentCardServices</ConnectionString>
<SQLQuery>
<RawSQLQuery>UPDATE FileNumber SET FileNumber = 1 WHERE ID = 'ActiveFGN'</RawSQLQuery>
</SQLQuery>
</TestStep>
</TestSetup>
<TestExecution>
<!-- WHEN: TWS calls BizTalk to trigger the validation of payments -->
<TestStep assemblyPath="Acme.BizTalk.Fx.Testing.dll" typeName="Acme.BizTalk.Fx.Testing.BizUnit.Wse2WebServicesClientProtocolStep">
<WebServiceUrl>http://localhost/Acme.CardProcessing.Secure/CardService.ashx</WebServiceUrl>
<ProxyTypeName>Acme.CardProcessing.Utilities.Testing.TestProxies CardService, Acme.CardProcessing.Utilities</ProxyTypeName>
<ProxyMethodName>SubmitBatchValidation</ProxyMethodName>
<ExpectingError>false</ExpectingError>
<InputMessageTypeName>Acme.CardProcessing.Utilities.Testing.TestProxies.CardValidationBatch, Acme.CardProcessing.Utilities</InputMessageTypeName>
<MessagePayloadPath>..\..\..\Acme.CardProcessing.AcceptanceTests\Features\BatchValidation\TestData\CardChanges.Input.xml</MessagePayloadPath>
<!-- THEN: The response will indicate that validation has started -->
<ValidationStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.XmlValidationStep">
<XPathList>
<XPathValidation query="/ValidationBatchesReturn/Return">0</XPathValidation>
</XPathList>
</ValidationStep>
</TestStep>
<!-- THEN: BizTalk will start the batch processing orchestration -->
<TestStep assemblyPath="Acme.CardProcessing.AcceptanceTests.dll" typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Batch validation started</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: BizTalk will send the batch to the acquirer -->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Batch sent to acquirer</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: The acquirer will find the batch in their input folder -->
<TestStep assemblyPath="Acme.BizTalk.Fx.Testing.dll" typeName="Acme.BizTalk.Fx.Testing.BizUnit.FileExistsStep">
<Timeout>60</Timeout>
<DirectoryPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</DirectoryPath>
<SearchPattern>*.*</SearchPattern>
<ExpectedNoOfFiles>1</ExpectedNoOfFiles>
</TestStep>
<!-- THEN: The acquirer will put the response in their output folder -->
<TestStep assemblyPath="Acme.CardProcessing.AcceptanceTests.dll" typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.ProcessAcquirerFileStep">
<DirectoryPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</DirectoryPath>
<SearchPattern>INZET.DANG.MDNDMDM</SearchPattern>
<ResponsesXmlFileName>..\..\..\Acme.CardProcessing.AcceptanceTests\GenericTestData\HSBCDefaultBatchResponse.xml</ResponsesXmlFileName>
<SchemaFileName>..\..\..\Acme.CardProcessing.AcceptanceTests\GenericTestData\BatchResponses.xsd</SchemaFileName>
<CreationPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerResponse</CreationPath>
<InvalidResponse></InvalidResponse>
<MultipleResponse>false</MultipleResponse>
</TestStep>
<!-- THEN: BizTalk will recieve an acknowledgement from the acquirer -->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Acknowledgement received from acquirer</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: BizTalk will recieve a response file from the acquirer -->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Acquirer response received</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: BizTalk will process any card updates -->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Processing card updates</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: The card will be updated in CARDSYSTEM-->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>CardServicesStub.CardServices.UpdatePaymentCards Called</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: The validation batch will be updated in CARDSYSTEM-->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>CardServicesStub.CardServices.UpdateValidationBatch Called</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: The card will be updated in CRM-->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>CRMApplicationStub.FinanceRefService.UpdateCreditCardDetails Called</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: Biztalk will have completed processing the validation batch response -->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Batch validation completed</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: The card update received by CRM will be valid -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileValidateStep">
<Timeout>6000</Timeout>
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CRMWS</Directory>
<SearchPattern>*.*</SearchPattern>
<DeleteFile>false</DeleteFile>
<ValidationStep assemblyPath="Acme.CardProcessing.Utilities.dll" typeName="Acme.CardProcessing.Utilities.Testing.BizUnit.XmlComparisonValidationStep">
<CompareTo>CardChanges.CRM.UpdateCards.xml</CompareTo>
</ValidationStep>
</TestStep>
<!-- THEN: The card update received by CARDSYSTEM will be valid -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileMultiValidateStep">
<Timeout>6000</Timeout>
<DirectoryPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CardSystemWS</DirectoryPath>
<SearchPattern>*.*</SearchPattern>
<ValidationStep assemblyPath="Acme.CardProcessing.Utilities.dll" typeName="Acme.CardProcessing.Utilities.Testing.BizUnit.XmlComparisonValidationStep">
<CompareTo>CardChanges.CardSystem.UpdateCards.xml</CompareTo>
</ValidationStep>
<ValidationStep assemblyPath="Acme.CardProcessing.Utilities.dll" typeName="Acme.CardProcessing.Utilities.Testing.BizUnit.XmlComparisonValidationStep">
<CompareTo>CardChanges.UpdateBatch.xml</CompareTo>
</ValidationStep>
</TestStep>
</TestExecution>
<!-- Clear out all test folders -->
<TestCleanup>
<!-- Acquirer folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerResponse</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<!—Card App folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CardSystemWS</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<!-- CRM folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CRMWS</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
</TestCleanup>
</TestCase>
This is just a reminder for my self of how and why we do this.
We have 2 projects within a TFS project collection for our integration component developments. We have:
1. A project for .net based integration projects
2. A project for BizTalk based integration projects
The main reason we do this is so we dont have loads of TFS projects as we have a significant number of components but also we want some different rules around check in and source control locks etc etc.....
At this stage our build servers are still running cruise control rather than Team Build so one of the problems i was finding was when cruise control labelled a build in TFS i couldnt tell easily in the label search which component or folder the build label related to. We modified the build scripts to do a custom label with TF.exe so that we could put a specific label from each build. We used the following command:
<Exec Command='"C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\tf" label "$(CCNetProject).$(CCNetLabel)" $(MsBuildProjectDirectory) /recursive'/"/>
Using this command means that the build label is applied at the right place for the workspace but also the label includes the cruise control project name making it easy to workout what the label was actually for rather than just being a bunch of version numbers.
There is probably a better way to do this but its working fine for us.
Wednesday, October 19, 2011
We are currently evaluating BizTalk 360 at one of my clients at the moment and I plan to write a more detailed post in the future about our experiences but in the meantime I wanted to make a comment about something this week which was particularly useful.
We have a very large project with many teams and vendors and in our first integration test environment all of the delivery teams do daily deployments to this test environment. From here a successfully tested release could be promoted to other environments as required. Its quite painful at times due to the pace of change in this first test environment and im sure most readers would be familiar with the typical challenges around integration and biztalk being the bit that identifies the problems and often not be the cause of the problem. Yes you know what I am talking about.
A significant amount of my teams time is taken up with troubleshooting issues and working out retrospectively where a problem came from. In our evaluations of BizTalk 360 one of the things I have done is to setup the BizTalk 360 health check emails to be published 4 times per day.
This week we had a situation where the test teams had signed off a release but had not realised for some reason there was an issue where some messages were unable to be loaded into an application. This whole release had been signed off to go into the UAT environment and then an hour later the BizTalk 360 email came out and we could see these suspended messages. We were able to identify and rectify this issue in the application which was not accepting messages and plug the gap in the test scripts before the release went to UAT.
Finding this issue in the system test environment and resolving it would have saved a significant impact with the problems this issue could have caused in the UAT environment in terms of lost test time etc etc. In my opinion this one situation probably paid for the cost of the BizTalk 360 license on its own for our large scale project so hopefully our evaluation will be finished soon and we will be able to roll this out across other environments.
Thanks Saravana keep up the good work
Friday, August 05, 2011
The biztalk videos for BDD and acceptance testing are now on cloudcast
http://www.cloudcasts.net/Default.aspx?category=BizTalk
Tuesday, July 19, 2011
I always seem to spend ages looking for this when i need it so this is the link to terminator
http://www.microsoft.com/download/en/details.aspx?id=2846
Monday, July 18, 2011
Ive recently done some user group sessions around BizTalk and Behaviour Driven Development and Acceptance Testing.
Ive uploaded the videos and samples for these sessions to the following codeplex site.
http://biztalkbddsample.codeplex.com/
They will also be on CloudCasts soon
Enjoy
Saturday, June 11, 2011
Ive just released an update to the codeplex project so that there are test steps which are also compatible with BizUnit v4 which has recently been released
http://btsloggingeventsinbi.codeplex.com/
Friday, June 03, 2011
I was chatting the other day with someone about adapters for connecting to LOB applications and an interesting point came up which I thought id share my thoughts on.
The scenarios is that if you have a line of business application for arguments sake lets say its dynamics CRM which has a BizTalk adapter available but also has an existing web service API (or some other protocol based API). Which should you use for integration? In my opinion the answer to this is the usual "it depends" answer. I think what is particularly important are some of the considerations which you should make when deciding your approach.
I thought id put down a few thoughts from my past experience and see where we end up.
1. Testability
In order to deliver a high quality integration solution its important to be able to test your solution very effectively. With some line of business applications if there is a standards based interface and an open protocol then its usually fairly easy to stub out the dependent system making it easy to test your solution in isolation.
If you have an application adapter then this can often create a partial coupling to the line of business system for your testing. There are techniques that BizTalk people can use to handle this such as changing to a different adapter for some BizUnit tests in the development environment to allow you to do this testing.
With a little up front thinking you can usually plan how your development process will work in relation to this dependency but I have seen projects where they ended up with a problem because the integration team could not work because of the dependency on a line of business application which was also in development at the same time and the application developers had broken something which was a blocker for the integration team.
The key point here is that a protocol adapter probably makes it easier to stub out your dependencies.
2. Dependencies
Application adapters will often require some kind of client side components to be installed on the BizTalk server. Protocol adapters are usually unlikely to have many client side requirements.
You should be aware of these dependencies and ensure you can install and manage them and that there are no conflicts with other software you need.
I remember once there was a project where we were integrating into both Cognos and Teradata and the client side dependencies of the vendor components we were using to connect to these systems were incompatible meaning the two integration components could not co-exist on the same server.
3. Vendor support
If your using a protocol adapter then in most cases you are probably using the normal out of the box BizTalk ones which will be part of your existing BizTalk support agreement. You will also be likely to have an agreement with the vendor for your application about the API they expose. In this case unless you have a less common protocol then for most cases your normal support agreements will cover things.
In the case of an application adapter then you need to consider who is the vendor for the adapter. If it's a Microsoft adapter then your probably in the same situation as with your normal BizTalk support agreement. If the vendor is a third party or the same as the vendor for the application then you would be likely to need additional support costs and considerations. In the case when the adapter vendor is a 3rd party you often will have a partner who is an expert in this kind of integration but also you could find it difficult to workout if a problem is BizTalk itself, the adaptor or the application.
The support position should be an important consideration.
4. Vendor Technology Road maps
Its important to be aware of the technology road maps for the application, BizTalk and any adapter vendor.
In the case of a normal protocol adapter there is probably only two parties involved and the communication would usually be standards based so it should be easy to workout any compatibility issues. You then just have the challenge of a contract-based interface to manage and any information about breaking behavior of contract changes.
The application adapter could be a much more complicated scenario. You need to think about the following:
- If I upgrade the LOB application will it work with my existing adapter
- If I upgrade my LOB application and need to update the adapter will this work with my version of BizTalk
- If I upgrade BizTalk will the adapter continue to work with the new version of BizTalk
I think in the application adapter scenario you need to have a much closer a alignment between the road maps in your organization for compatibility of BizTalk, the adapter and the LOB application.
5. Current Organisation Approaches
You should think about if your organization already integrates with the application in a certain way from outside of BizTalk. If it does and there are similarities then you can have a certain degree of confidence in what to expect. You will also have in place some processes to manage this.
If you find your organization has an existing way of doing this that works well and fits with the strategy then definitely follow it. If the existing way doesn't work then definitely look to change it.
6. Cost
Adapters which aren't free from Microsoft or included with BizTalk are likely to cost something.
This is an obvious factor to consider. But also consider the costs of not buying an adapter. It's much harder to quantify something you build but the long term cost of ownership may be much higher than the original vendor purchase.
7. Assumptions
When you connect to an application you are making assumptions about the contract based on something like wsdl if it's a web service interface. You are also making assumptions about the behavior of the API usually based on the documentation which comes from the application vendor.
Typically when we integrate with an application using a protocol adapter I would normally look to develop some tests, which will flex the API without any of the stuff I will build on top of it. These tests will really prove my assumptions about how the application will work and potentially give me an indication of performance. When I am using a protocol adapter this kind of test is usually relatively easy to write in .net code even if the protocol adapter ends up being something from a 3rd party.
When I use a BizTalk application adapter it is likely to be a little bit harder to validate these assumptions. I can still do it but I will have to build a fairly simple BizTalk application which will allow me to push some messages through the adapter and prove how the interface will work. Its usually just more time consuming to develop tests in this way.
One point to note is that if the adapter is a WCF LOB adapter then it is often possible to use this adapter outside of BizTalk so you can potentially write the same .net tests mentioned above but use a WCF LOB adapter directly from .net code.
The key thing with this section is that it's a good idea to prove how the interface will work before you start using it so you can reduce the integration issues and mitigate the risk that you could have problems with the assumptions you make about the vendor API.
8. Feature set
With a BizTalk application adapter there is usually a reason why they have been built. It is important to be clear on the feature set offered by the adapter and what it offers above just using a protocol adapter.
In some circumstances you will have the situation where a BizTalk adapter was built and then a new version of the LOB application comes out and part of this new version includes an improved API which is more standards based and other good things. The adapter may still be compatible.
In other situations there will be clear features offered by the application adapter which you would either have to develop yourself or things which just improve the integration experience.
Carefully evaluate the features offered by each adapter to workout what you get extra and how valuable it is.
9. Buy Versus Build
Some organisations have a clear buy versus build strategy. Without going into too much detail this could significantly influence your decision.
10. Be Open Minded
One of the things that can be quite frustrating is when a design decision is made simply because that's the way its always been done and there is fear of change or something that the team do not understand. Just because you always use a certain way to connect to a given application doesn't mean a different way does not have its own merits and might not be better. Particularly if a new version of an adapter comes out.
On this point I would say don't always jump to your default choice. At the very least try out the other options.
Summary
As I said at the start of this article there is no one answer for every situation, but in practice I often find that the application adapter could be more work or cost to your solution. In this case I usually want to have clear justification for why we are using the application adapter over the protocol adapter so that we have a well defined design decision for our project which is also inline with our integration strategy.
This was a bit of a brain dump from my experience and it would be interesting to see if others agree or disagree with my thoughts.
Tuesday, May 17, 2011
A few weeks ago I did the video about how we were testing what happens inside BizTalk by using the CAT Team Logging Framework and the ETW trace events and then testing the information coming out from our tracing to prove what was happening inside BizTalk.
Ive updated the codeplex project so that the code is a little easier for people to use and is packaged better. Also the source code is now available in codeplex and there is some additional documentation.
Note that it now works slightly differently to in the video but performs much better. The original video about the idea is on the following link
http://www.cloudcasts.net/ViewWebcast.aspx?webcastid=2521028145037920619
The codeplex project is on the following link:
http://btsloggingeventsinbi.codeplex.com/
Friday, May 13, 2011
Ive recently just setup BizTalk 360 (http://www.biztalk360.com/) to try out the monitoring capabilities. I also thought it would be interesting to see how things go since this given environment was a BizTalk 2006R2 environment and fairly old stuff now.
I had a couple of issues during the install/setup but i was able to solve all of these using the troubleshooting page on the BizTalk 360 website and a little common sense. To be honest a couple of them were me being a bit lazy and not reading the install notes properly.
The changes I had to make were as follows:
1. Changed connection string to the biztalk 360 database as described on the troubleshooting website to integrated security
2. Change the IIS6 virtual directory to .net 4 rather than .net 2
3. Add the full trust element to the web.config file for BizTalk 360
4. Add the IIS NT Authentication providers as described on the troubleshooting page
Bearing in mind this was a CTP release I was very pleased with the support on the website which helped me to get this setup and even with these problems it still took less than 20 minutes. I would expect BizTalk 2006 to be a slightly less target for the BizTalk 360 team than BizTalk 2010 but it goes to prove it can be installed pretty easily and im not enjoying the management features which I think are a big opportunity for lots of organisations to really get their production operations of biztalk working in secure and effective way
Good job guys
Saturday, April 23, 2011
Ive eventually had time to migrate the build generator to work with BizTalk 2010.
Ive released this as a beta for now and would love to hear any feedback from others who might be using it while i work on getting some more testing with other migrated projects.
If your interested in taking a look please refer to:
http://biztalkmsbuildtool.codeplex.com/releases/view/65036