Tuesday, August 12, 2008
Had one of those days yesterday and today, you know what its like when you find a bug/problem and you dont really get a chance to properly look at it when your doing a million things at once.
It is so often that the cause of the problem is really obvious but you just cant see the wood for the trees.
This is a reminder for myself as much as anything.
I was writing an msbuild task to wrap cruise controls ICruiseManager so I can get the latest build label of a project that has been built.
The problem was that when I tested the task using MsTest from Visual Studio it worked fine, but when I ran it from an MsBuild script it kept failing complaining about not being able to find the ThoughtWorks......Remote assembly. I ensured that the assembly was where my tasks assembly way and frustratingly it wasnt working
When my brain returned I spotted that the problem was that the assembly is being searched for in the directory which the process started (the .net framework directory where MsBuild.exe is) so by copying the assembly there it now works.
All of the slide decks from last months UK SOA/BPM User Group session are now available from our site
http://sbug.org.uk (in the media section)
The decks covered were:
Oslo - Robert Hogg
Composite Applications - Andy James
SOA in 2015 - Darren Hallett
Just been reading the quarterly newsletter from Frends (http://www.frends.com/), it was nice to see a little plug for my recent article about scheduling requirements for BizTalk solutions where I discuss the use of their product as one of the possible ways to implement this kind of requirement:
http://geekswithblogs.net/michaelstephenson/archive/2008/05/16/122203.aspx
Tuesday, August 05, 2008
From previous posts you may have read how I integrated the Microsoft BizTalk documenter into my MsBuild process.
In general this has been working fine, but every now and again I kept getting a build failure when trying to generate the documentation as follows:
System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Documents and Settings\<UserAccount>\Local Settings\Temp\BTS2K4Doc\Application\fb7b6ba7-016c-4a1f-9c7d-ec9037911456.html'
If you get this it seems to be caused by an hhc.exe (Microsoft Help Compiler) process still being alive from a previous run. Just kill it and the builds will work again.
Tuesday, July 29, 2008
Article Source: http://geekswithblogs.net/michaelstephenson
Recently at the UK SOA/BPM User Group Yossi Dahan and I chatted briefly about using MsBuild and BizTalk, more specifically Yossi mentioned that he wished it was possible to regenerate the schemas for consuming a service automatically.
Following this discussion and a few other things I had a blast at seeing how difficult this would be to do. I think there is a lot of value in having this approach as it fits well with a contract based development approach and continuous integration. I want to be able to regenerate my schemas every time I build the solution, and if a breaking change has been introduced I want the solution to break so it is corrected.
This fits with what I believe is a good practice to identify these kinds of problems as early as possible in the development cycle.
The aims of the task are:
- Act in a similar way to the Generate Schema feature for BizTalk/Visual Studio except that it will be driven by an MsBuild task rather than a GUI
- Just deal with the schemas, I wasn't too bothered about regenerating the orchestration and binding file samples
(The sample is available for download at the bottom of the article)
The Sample
In the sample the quickest way to demonstrate this is as follows:
- Modify the BuildProcess.xml file to point to your url for the MexEndpoint
- Double click the RunBuild.cmd file
- Check the Acme.BizTalk.Schemas.WCFService folder and you should see your schemas have been updated
It is expected that you would include the schemas in your project, but exclude them from source control so they can be rebuild every time you build the solution.
Digging Deeper
I managed to work out how to do this by inspecting the Microsoft.BizTalk.Adapter.Wcf.Consuming with Reflector. Upon investigation it would have been nice if a few of the classes were externally available from the component. As it was I needed to disassemble the component to progress this idea. In the disassembled component I created a new version of the Consumer class in which I removed some of the functionality and changed the Consume method interface so it wasn't dependant on being passed a Visual Studio Project object.
With a couple of other tweaks I was able to get this working
Conclusion
Although this task seems to do the job, it has not been extensively tested, and it also only currently works with the MEXEndpoint and not yet with meta data files.
Hopefully this idea will help a few people, and the idea might get back to the product team as a possible future enhancement as this kind of thing can certainly help your development process be more effective.
Saturday, July 26, 2008
Article Source: http://geekswithblogs.net/michaelstephenson
A few weeks back I got a comment to one of my blog posts by a guy who said he wished he had more control over persistence points in an orchestration. In his example he basically wanted to reduce the number of persistence points as he needed to improve performance in what sounded like a request response scenario.
Often in these kinds of scenarios if BizTalk has been well optimised and you still do not get the latency requirements you need then the likely hood is you are trying to solve a problem with BizTalk when a different solution would be most appropriate.
Anyway what did make me wonder was the opposite of what this guy wanted to do. My thoughts were:
- Is there a situation where I would actually want to persist the state of an orchestration myself?
- If I wanted to do this how could I do it?
- If I can actually do it where might I have used it?
- Would we benefit from an explicit "save" shape?
Anyway when you travel around a lot like myself you tend to get plenty of boring delays in the airport so on one such occasion I had a crack to see if you could do this.
How would I do it?
I wanted to do the simplest example which could demonstrate how I could persist the orchestration state from C# and then do something to show how if an error occurs and the orchestration suspends then I can resume it and it will complete as expected.
In the example orchestration I will receive a message which I won't really do anything with, it is simply to start the process. From here I will initialize a counter and then perform a loop from 1 to 20. Normally I will just trace out the current ordinal, but on the 10th iteration I will cause an error to be thrown (I simulate this by checking if a file exists on disk and if so throwing an error). When the orchestration suspends it can be resumed manually and it will not throw an error when it replays the 10th iteration.
After 20 iterations it will exit. The picture below shows the orchestration.
Now normally in the above orchestration, because there are no persistence points within the loop you would get an output in DebugView like in the below picture.
Here you can see after the error is thrown the process will restart from the beginning again. The next picture below shows what happens when I persisted the orchestration at each iteration in the loop. You can see how the process was resumed and started by replaying iteration 10 and continuing to the end.
Ok so for most people there is nothing new here, but that is just background information. The next thing is in this example is because I did not use any of the normal shapes that cause the state to be persisted how did I achieve this?
Within the Microsoft.BizTalk.XLANGs.BTXEngine namespace there is a BTXService class. With the below line of code I was able to persist the state of the current orchestration.
BTXService.RootService.Persist(false, BTXService.RootService.RootContext, false, false, false, false);
Thoughts now
Ok so I have been able to do what I was wondering about, but as you can see to do it you have to use some of the "under the hood" BizTalk components so it isn't really a recommended way of doing this. This is another just because you can doesn't mean you should thing.
I was discussing this with some friends in the pub a while back and between us our thoughts were:
-
None of us could think of a specific example of where we would have used this technique, so that probably means we don't need a "Save" shape
- If we were doing something like this we would possibly have it within an atomic transaction scope anyway so it would create a persistence point before our work anyway. We kind of thought that this was one of the benefits of the XLANG engines approach to allowing you to develop orchestration workflow by protecting the developer from things such as this where if you can do too much you would be likely to get yourself into trouble.
I'd be interested to know if anyone has come across a situation where they felt this feature to explicitly persist the orchestration would have helped.
If you are interested in the example it can be downloaded below:
Friday, July 25, 2008
Article Source: http://geekswithblogs.net/michaelstephenson
I have often come across situations where I have been asked to look at a process (usually in BizTalk) where it isn’t quite running as the customer would like. I have decided to start a series of posts which I will call refactoring tales. These posts will discuss a the process implementation and the problems encountered along with it. I will then discuss the approach taken to improve things and what the benefits were.
Background
This particular process had been implemented as a sort of splitter pattern. Each day a file would be received which was then to be split and loaded into two different systems. The split was made by checking each record against a third system which would indicate which system the record should be loaded into.
This example is relatively common and the implementation which was developed was as follows:
1. The file was received as a complex XML file via the FTP adapter which had a service window applied to it A normal file would contain about 5000 records and was approximately 10MB. On exception there have been files as large as 40MB.
2. A pipeline component was applied in the decode stage which would remove a pre-processing instruction from the file
3. The file was delivered to the Messagebox
4. An orchestration would subscribe to the message and start processing
5. Some validation logic is applied to ensure the message is in sequence. The logic involves using the SQL adapter to check a database to ensure the message is in sequence.
a. If the message is not then an event is logged and a manual recovery process is initiated and the orchestration suspended
6. In the orchestration it would iterate through records in the file extracting each record.
7. In each iteration of the loop the extracted record is checked against an external system to see where to send it by using the oracle adapter
8. If the message is sent to system A then a web service call is made using the WSE 2 adapter to load the message
9. If the message is to be sent to system B then an in memory message is built up for all of the messages for system B which will then be sent in batch
10. When all or the records have been processed then the records for system B are send via FTP to that system. The batch message (about half of the original file) is transformed in the send pipeline to a flat file format.
11. The batch management database (a custom database which is very simple and to support the sequence number validation) is updated to mark this batch complete
12. Finally an email is sent to the business users to confirm the results of the batch processing (how many went to each system)
The Pain
The following pain was experienced with this process:
Length of time it runs
Each record in the file would take approximately 4-5 seconds to process with a little time before and after the loop. It was common for the file to take in excess of 6 hours to fully process. The result of this was a file received in the morning would often take almost the whole business day to be loaded.
Cannot scale
One of the key limitations of this design is that because the process is sequential it cannot be scaled out. If the file were to suddenly become 10,000 records you would expect the processing time to double. Adding more BizTalk servers to the group would have no impact on this duration.
Does not recover well from errors
During the first use of this service there were some network connectivity issues and this ment that sometimes the 3 retries on the web service port would all fail. This meant that the exception was caught and the process stopped until someone looked at the problem. When resolved the orchestration was resumed it would continue. But as you can imagine there are two significant impacts on the amount of time the overall process would take.
The first was the response time from a manual operator. This could add anything from a few minutes to an hour that the whole process would be stopped.
The second was that each port error would initiate one of the port retries about 5 minutes later. In the example here about 10% of the calls failed (250+ because not all of the records went to system A) and you can imagine the impact of these extra minutes on the overall processing time. One file took 2.5 days to fully load.
Persistence Points & SQL Transaction Log Backups
One of the behind the scenes problems was that because of the fact that the orchestration was holding one large message in memory from the start and building up another through the execution of the process it meant that every persistence point was expensive with the orchestration state being between 10-20 MB for the standard message size.
There were approximately 2 persistence points per record so for a standard file (5000 records) that means 10000 persistence points. If you do the calculations here you can see we are persisting somewhere in the region of 100Gb worth of data to the messagebox during the processing of this 10 MB file just to save the state of the orchestration.
The SQL Transaction log backups for the messagebox were eating drive space like crazy.
There were a couple of other smaller things too, but as you can see there is a clear business and operational need to refractor this.
There were a couple of lessons to learn from this situation, particularly around performance testing and the difference in how a process will be perceived to run on a developer machine versus how it will run in a live environment, but I will not go into these within this post.
The Refactoring Challenge
Refactoring is different to re-writing. If we were doing this from scratch there would be a number of different things you could do, but one of my favourite things about these kind of situations is there are things you can change and things you can’t. The challenge is to make it work much more effectively while minimizing change/cost. Some of the factors which affected how I could refactor this are as follows:
· No change to external systems
In this case I could not change any of the external message formats or communication mechanisms. There was no opportunity to change the external systems functionality.
· Short timescale
I only had a short amount of time to POC a new idea for this and then it would be implemented by other developers.
· The business users still want their email
The email feature for the users screams out as a BAM opportunity. However in this refactoring there is no opportunity to change this or retrain users on how to use the BAM portal etc.
· Limitations on technologies to use
This particular solution could potentially use SSIS to help with the implementation, in this particular case it was preferred to keep it as much in BizTalk as possible to keep the supportability of it within what had already been trained to the support team.
The New Implementation
After weighing up all of the factors and components I had to work with I decided that the best way to refactor this process would be as follows:
1. The file was received as a complex XML file via the FTP adapter which had a service window applied to it
2. A pipeline component was applied in the decode stage which would remove a pre-processing instruction from the file
3. An XmlDisassembler was added next in the pipeline to promote some metadata about the batch from the message header to the message context
4. A new custom pipeline component was added to the pipeline to rip apart the batch.
This new pipeline component would use the XPathReader to read the message and to extract each record from the message. The database which was part of this solution used to manage batches was extended to have a table to contain a temporary storage for the batch records. The pipeline component would save all of the records to the database and then change the message which would come out of the pipeline component to be a very small trigger message which would be sent to the messagebox and contain the ID of the batch and the number of records it contains.
5. A new orchestration would subscribe to this trigger message and start a controller orchestration which would manage the overall process
6. The controller would iterate through a loop from 1 to the number of records in the batch and create a small simple command message which would be sent to the messagebox with an instruction to process a specific record from this batch
7. The controller orchestration would then sit polling the database until all of the records have been processed
8. For each of the command messages which had been sent to the messagebox an instance of an orchestration would be initiated which would process that particular record. If there were 5000 records you would get 5000 instances of the Record Processor orchestration
9. In the Record Processor orchestration the message it receives will tell the orchestration which batch and record number to process so it can get its data from the temporary database. This orchestration will then call the system which helps decide if the record is sent to system A or system B
10. If the record is sent to system A then the web service is called. The orchestration will then update the temporary database to indicate where the record is intended to be sent to.
11. When all records have been processed the Controller orchestration will detect this and stop polling the database.
12. The controller orchestration will extract all of the records to be sent to system B and then pass this to the messagebox so it can be mapped and delivered to system B via a port subscription.
13. The controller orchestration will query the temporary database to get the values to create the email which advises the business users how the batch was split and send it via the SMTP adapter
14. Finally the controller orchestration will clean out the temporary storage of the batch and update the sequencing ready for the next batch
Some additional points to note here are:
· If the receive pipeline gets an error then the batch would stop saving to the temporary storage. If the instance was resumed then the import would begin again from scratch
· I considered using a normal BizTalk debatching approach but encountered problems once the batch size got above 1000 records (can’t remember the error message now). Based on this I chose to implement a custom debatching solution
The Benefits
As result of the above refactoring the benefits were:
· Because the records were being processed in parallel I have been able to process a 10MB normal size file in around 15 minutes compared to the previous 6 hours
· Any errors with one record do not cause a backlog of all of the other records
· The improved performance means all of the records are loaded before the business users start their working day rather than the previous solution where they were almost finished the day before the records are all processed
· Because the messages being processed in the orchestrations are smaller than before (approximately 2kb) the fact that overall there are more persistence points (about 4 per record now) means that the total data persisted to the messagebox to maintain the state of the orchestrations has reduced from somewhere in the region of 100GB to < 40MB
Conclusion
Hopefully this post will give a real world example of how being pragmatic in a refactoring exercise can help you make significant improvements to a process with minimal cost to the project. This article may be a little unclear in how this has been implemented so if there is some interest in it the way I implemented this is fairly generic so I could potentially do a future post with an example.
Wednesday, July 23, 2008
I came across an annoying one the other day, I havent had time to look into it in more detail, but here are some notes about it.
In some of the tests we do with BizUnit we sometimes check the event log to see that certain messages have occured. I came across an example I havent noticed before the other day.
In my test I do a bunch of stuff then I wait until a custom event with a specific Event Id is logged to the event log. I use the BizUnit event log check step and it finds my event fine.
Later in the test I will do the following:
- Stop the application pool for a web service I will call
- Allow BizTalk to call the service and then suspend the service instance because the web service is unavailable
- My test will detect the event logged by BizTalk Event ID - 5754 to indicate a call has failed. I will also use the ValidationRegEx node to confirm the event message relates to my port
- I will then start the application pool back up
- I will then use a custom BizUnit step to resume the suspended instance
- Finally I will check the process worked correctly
This was working as far as the point of detecting the BizTalk event. I could see in the event log the event I was expecting was there, but for some reason my step didnt spot it.
Because I was using the regex feature this was my first thought, but the expression was simple and even if I took it away the event wasnt spotted still.
It turns out when I wrote a little C# to check what comes back in the event log was that although the event in Event Viewer displays with the event number 5754, the event log instance from System.Diagnostics comes back with the following property values:
- Event Id: 12588666
- Instance Id: 3233814138
When I changed the event number as appropriate it worked fine. Frustrating but there you go.
Wednesday, July 16, 2008
On one of the projects im associated with we have recently upgraded from BizTalk 2006 to BizTalk 2006 R2 for our next release.
In doing this we needed to upgrade our development environment and a number of testing environments. Fortunately the testing envronments are not managed by our team so we didnt need to have too much involvement in this other than providing guidance and support.
In the development environment we needed to upgrade a number of developer machines and also some build servers. We also needed to support some cycles of testing which were still happening on the non R2 machines.
Two things which you might want to consider if you are doing the same is as follows:
1. The importance of Continuous Integration
One of the first things I did was to setup the appropriate build servers to implement continuous integration so we were building and testing our code on both BizTalk 2006 and 2006 R2 at the same time.
One of the benefits of this was that it allowed me a chance to be able to deploy code which was tested in both environments and to give us more time to rebuild the developer machines.
In theory there shouldnt really be many issues for this upgrade in terms of the code we had already implemented, but using CI would help us mitigate the risk.
2. POC
Before we had begun the upgrade I had setup a machine to POC running our code in a 2006 R2 environment. Although we expected this to be trouble free I did find two small problems which needed addressing (1 a BizTalk thing and one not)
Problem with a custom pipeline component
The first problem I came across was in a custom pipeline component we have. This component helps implement a custom debatching solution in a process where we implement the splitter pattern (i will probably post something about this in the future). This component basically accepts a big message comming in, does some stuff and then replaces the message with a small trigger message which is sent to the messagebox.
The message we send out matches a schema and we set some custom context properties and the appropriate message type property. This message is then collected by an orchestration and the process continues.
While this had worked fine previously in 2006. When I ran this in BizTalk 2006 R2 I would get an error from the orchestration when it collects the mesasge and initiates the orchestration. The problem indicates that the schema strong name is incorrect.
I have come across this before and by changing our pipeline component to add the schema strong name of the message to the context (BTS.SchemaStrongName) the whole thing then worked again.
This is obviously caused by something that has changed between versions and was a simple enough fix.
Problem with one of our tests (Serialization)
The second problem was also pretty straightforward. When running the tests for one of our applications the SoapHttpRequestResponse step from BizUnit started to get an error which it hadnt previously got. This was caused because there must be some changes in the serialization functions of the ,net framework (we now have .net 2 SP1, .net 3 SP1 and .net 3.5, where we used to only have .net 2). Basically the problem was that the developer hadnt included the namespace as defined in the WSDL. While it used to work when the step used it, it now didn't
In conculsion so far the impact on our code by this upgrade has been very small and the couple of problems we have had have been eaily solved.
The one takeaway is probably my to use CI and BizUnit as it makes it much easier to detect problems before you deliver code outside of your team for testing.
Friday, June 20, 2008
The details of the first UK SOA/BPM User Group meeting are available at the following link
http://sbug.org.uk/forums/t/52.aspx
Sunday, June 15, 2008
This article is intended to explain how to use MsBuild to help you when using BAM on a BizTalk project.
When you use BAM one of the positives is that you have BM.exe and BTTDeploy.exe which can be used to help you do this deployment etc. However as I have mentioned in previous posts I feel one of the things that is a barrier to helping people get up and running with BAM is that there isn’t really anything (that I’ve seen anyway) which helps you to explain how your BAM work would fit into your normal solution development build and continuous integration processes.
To try to address this barrier hopefully this article will provide some information to help developers get over this.
I have extended the tasks within my BizTalk MsBuild Generator project on codeplex(
www.codeplex.com/BizTalkMsBuildTool) to have some additional tasks for BAM. These tasks are nothing special and simply provide a nicer interface to BM.exe and BTTDeploy.exe. The tasks will be published in the next release, but you can use them now by getting the sample below or contacting me.
The tasks I have created are:
|
Task Name
|
Description
|
Example
|
|
BackupAllDefinitions
|
This will backup all definitions in BAM. I find this is useful so incase you happen to do something wrong you can clean the whole of your development BAM and redeploy it as it was from this backup file
|
<BizTalk.BuildGenerator.Tasks.BAM.BackupAllDefinitions
OutputFilePath="FullBAMBackup.xml"/>
|
|
BackupDefinitionFile
|
This will back up the definition file in case you have a problem and need to restore it to before the most recent build
|
<BizTalk.BuildGenerator.Tasks.BAM.BackupDefinitionFile
DefinitionFilePath=”MyDefinitionFile.xsl”
OutputFilePath=”MyDefinitionFileBackup.xsl”/>
|
|
BackupTrackingProfileFile
|
This will back up the tracking profile file in case you need to restore it to the last version
|
<BizTalk.BuildGenerator.Tasks.BAM.BackupTrackingProfileFile
TrackingProfilePath=”MyTrackingProfile.btt”
OutputFilePath=”MyTrackingProfileBackup.btt”/>
|
|
CleanBam
|
This will backup all definitions in BAM and then remove everything from BAM
|
<BizTalk.BuildGenerator.Tasks.BAM.CleanBam
TempDefinitionFilePath=”BackUpFile.xml”/>
|
|
RemoveTrackingProfile
|
This will remove a tracking profile from BAM
|
<BizTalk.BuildGenerator.Tasks.BAM.RemoveTrackingProfile
TrackingProfilePath="FileProcessBackup.btt"/>
|
|
RemoveDefinition
|
This will remove a definition from BAM. It will remove all activities and views defined in the .xsl or .xml file
|
<BizTalk.BuildGenerator.Tasks.BAM.RemoveDefinition
DefinitionFilePath="BamWorkbookBackup.xls"/>
|
|
RemoveActivity
|
This will remove an activity with a specific name from BAM
|
< BizTalk.BuildGenerator.Tasks.BAM.RemoveActivity
ActivityName=”MyActivity”/>
|
|
RemoveTrackingProfile
|
This will remove a tracking profile from BAM
|
<BizTalk.BuildGenerator.Tasks.BAM.RemoveTrackingProfile
TrackingProfilePath=”MyTrackingProfile.btt”/>
|
|
RemoveView
|
This will remove a view with a specific name from BAM
|
<BizTalk.BuildGenerator.Tasks.BAM.RemoveView
ViewName=”MyView”/>
|
|
DeployDefinition
|
This will deploy a definition .xsl or .xml file to BAM
|
<BizTalk.BuildGenerator.Tasks.BAM.DeployDefinition
DefinitionFilePath=”MyDefinition.xsl”/>
|
|
DeployTrackingProfile
|
This will deploy a tracking profile .btt file to BAM
|
<BizTalk.BuildGenerator.Tasks.BAM.DeployTrackingProfile
TrackingProfilePath=”MyTrackingProfile.btt”/>
|
|
UpdateDefinition
|
This will update a definition be using the update command and supplying the updated .xsl or .xml file
|
<BizTalk.BuildGenerator.Tasks.BAM.UpdateDefinition
DefinitionFilePath=”MyDefinition.xsl”/>
|
Before I get into the details of the sample, some of the factors I have considered when demonstrating the process in the sample are:
· You may have more than 1 application running/building which would need to use BAM so in most build processes you would probably not use the CleanBam task as this may break another build
· I tend to use the backup tasks to help me because in the past I have had a few problems where I have changed for example my definition file and then couldn’t remove the definitions because the .xsl file had changed too much.
o To get around this I use the backup everything before I start interacting with BAM so I can get back to the previous state if it goes wrong.
o At the end of the build I back up the definition and tracking profile files so that if I do change them before the next build I can still remove what is there.
o When I remove from BAM I usually specify the paths to the backup files
o In addition to the above you could also use the RemoveView/RemoveActivity tasks but I find this is a little more tedious as I have to remember to modify the build when I change the definition
The Sample
In the sample (available below) I have created a basic BizTalk project which just moves a file from one location to another through an orchestration. I have also created a BAM definition for it and also a tracking profile. You should be able to see these within the solution.
For the sample I don’t really want to focus on the details of the BAM definition or tracking profile, but rather the deployment and build scripts. In the build process for this application I have used the BizTalk MsBuild Generator to create the MsBuild scripts for me.
In the Acme.BizTalk.BamDemo.Custom.targets file I have extended the build process in the normal way with these scripts to add my BAM functionality. The rest of this section will discuss how I have done this.
Note: To get the build running on your machine you may have to change the following:
1. There is a property defined in the Acme.BizTalk.BamDemo.properties file to indicate the location where you place the source code
<SourceCodeRootFolder>C:\Blog\MsBuildBAM\Acme.BizTalk.BamDemo</SourceCodeRootFolder>
2. The bindings file may need changed to point to the correct location where you put the input and output folders.
Cleaning up BAM
In the build script I may have to deal with previous builds and deployments which have already setup and configured BAM to know about my implementation. In order to run a consistent build process it is important to clean up after any previous build. To do this in the BeforeCleanUp target I will call some of the BAM tasks to clean up BAM. The below code snippet shows that I am backing up BAM as described above, and then removing my Tracking Profile followed by my Definition.
<Target Name="BeforeCleanUp">
<BizTalk.BuildGenerator.Tasks.BAM.BackupAllDefinitions
OutputFilePath="FullBAMBackup.xml"/>
<BizTalk.BuildGenerator.Tasks.BAM.RemoveTrackingProfile
TrackingProfilePath="FileProcessBackup.btt"/>
<BizTalk.BuildGenerator.Tasks.BAM.RemoveDefinition
DefinitionFilePath="BamWorkbookBackup.xls"
ContinueOnError="true"/>
</Target>
Deploying to BAM
In the build script I also want to deploy to BAM to setup the details of my tracking requirements. To do this I have chosen to extend the AfterApplyBindings target. You can do this in a few places in the build script so long as you ensure you have deployed your assemblies to the GAC and management database so they can be resolved when applying your tracking profile. You can see in the snippet below that I am using the two deployment tasks and also the two backup tasks to ensure I always have a copy of what was deployed in case I make changes that I have problems then removing or undoing.
<Target Name="AfterApplyBindings">
<BizTalk.BuildGenerator.Tasks.BAM.DeployDefinition
DefinitionFilePath="BamWorkbook.xls"/>
<BizTalk.BuildGenerator.Tasks.BAM.DeployTrackingProfile
TrackingProfilePath="FileProcess.btt"/>
<BizTalk.BuildGenerator.Tasks.BAM.BackupDefinitionFile
DefinitionFilePath="BamWorkbook.xls"
OutputFilePath="BamWorkbookBackup.xls"/>
<BizTalk.BuildGenerator.Tasks.BAM.BackupTrackingProfileFile
DefinitionFilePath="FileProcess.btt"
OutputFilePath="FileProcessBackup.btt"/>
</Target>
Summary
Hopefully this article shows that BAM is not as difficult to get up and running as you might think and this will encourage some development and project teams to use it more in their projects. The sample is available below:
Saturday, June 14, 2008
Introduction
I did this article ages ago and have just found where I had misplaced it.
Anyway the intention is to provide some introductory information about BAM (Business Activity Monitoring) for those who are relatively new to BizTalk and to help people to decide if it is suitable for their project. I will also provide some links to useful resources I have used before when working with BAM.
What is BAM?
Business Activity Monitoring (BAM) is a feature of BizTalk which allows you to obtain information and metrics around your executing processes. To do this you define an observation model of your process which indicates which metrics you are interested in. This information is then captured during the execution of the process and can be queried by business users. BAM has a number of tools which lets different types of users interact with this view of the process.
|
User Type
|
Tool
|
What do they do
|
|
Business Analyst
|
Excel Add-in
Orchestration Designer for Business Analysts
|
These tools help the Business Analst to define the data to capture and to define how the data will be aggregated
|
|
Developer
|
Tracking Profile Editor
BAM API
BAM Web Services
|
These tools/components allow the developer to map the observation model to the physical implementation of the process. An example of this would be indicating that a particular shape in an orchestration maps to a mile stone defined in the observation model
The BAM Web Services allow a developer to integrate custom applications with BAM so they can query inform from BAM
|
|
Business User
|
BAM Portal
Excel
|
These tools allow a business user to access BAM information in tools which they are used to such as Office and Sharepoint
|
For a newbie some of the common questions which might be asked are:
What's the difference between BAM and HAT?
BAM is intended for business users to monitor business processes. HAT is intended for administrators to monitor and track the health of the system.
What's the difference between BAM and BAS?
· BAS provides information workers with features in tools such as Office/Sharepoint which allow them to interact with and manage processes involving Business Partners.
· BAM allows business users to access information about processes which have executed or are executing and aggregations of this information
Why use BAM?
The following are some of reasons why a client would like BAM:
|
Reason
|
Description
|
|
Business Knowledge
|
Unlike application development, BizTalk development often does not have anything tangible which can be shown to a client. BAM will provide them with important KPI related information about their business processes and can help them to see the benefits of the solution and help them confirm they are getting a satisfactory ROI.
|
|
Business Knowledge
|
A business user would be able to see real-time information about their business processes enabling them to be agile and responsive to ensure they are getting the most from their business. An example of this might be seeing that we are starting to build up a back log of orders in the warehouse so they could draft in more resources to help clear this temporary bottleneck.
BAM alerts can also provide the trigger to automate some of these decisions.
|
|
Cost
|
BAM is a built-in feature of BizTalk which a company who invests in BizTalk will already have made. The cost of mining information about processes with BAM would be significantly cheaper than using a third party tool. Although I’m sure it would be possible to capture the information in BAM and then export it to third party tools if required.
|
|
User Experience
|
BAM as a reporting solution provides a familiar environment to business users by taking advantage of Office and Sharepoint
|
|
Simplicity
|
BAM can be implemented with no code in lots of cases
|
|
Phased implementation
|
One of the cool things about BAM is that often a project chooses not to implement it as a core requirement, but an organisation can always come back to their established BizTalk solution and look to add BAM as an additional layer to the solution where they could make the information about processes available without having to deploy a new version of their BizTalk components which are already running. You would simply deploy the BAM artefacts to the existing BizTalk group.
|
How do I spot an opportunity for BAM on my project
1. If a client is considering BizTalk against a potential other product a demonstration of how easily BAM can be implemented in the solution and the valuable information that it can return can be a good way to promote the benefits of BizTalk.
2. If a client is considering a BizTalk solution but is also considering a custom approach which is a common case due to the cost of BizTalk, then it can be useful to discuss how the custom solution would be monitored and how the data would be made available in a consistent manner. This kind of thing would quickly increase the complexity of the custom solution and would make the fact that BAM is available out of the box and easy to implement into your processes an attractive proposition.
3. Although it is recommended to begin thinking about BAM from the start of your project BAM can be added to existing BizTalk solution later. This provides the opportunity for additional work on a project where BAM was not originally in scope. When you hear a client wanting to have a better understanding about their business processes or when you hear them wanting to have volumetric type information relating to their processes this is an opportunity for us to talk to them about adding BAM to their existing solution.
4. If you find that people struggle to understand the role BizTalk is playing in a solution or thing that BizTalk is a magic box which makes things happen and people just generally don’t understand what it does then BAM can be a good way to give people visibility to what BizTalk does in a way they might understand.
5. If you are a consultancy then BAM is a fantastic opportunity for you to sell another project into your existing customers if BAM was not an original requirement of the other projects. You will already know their processes and if you can demonstrate the value of the information you can get your client then you have another potential project here.
Sometimes I think the best way to approach this is to take an existing process which you thing would benefit from being BAM enabled and then just do it as a POC. Showing the business the value of the information and how easily it can be accessed combined with the low cost involved in getting it is probably the best way to get people interested in BAM. I think it is common in these kinds of situations that when a business user sees something like this that will help them they will often come back and ask for more.
Components of BAM
This section will provide some terminology and basic information about some of the background components which make up BAM:
BAM Portal
The BAM Portal is an ASP.net web application which allows business users to view the information held in BAM. In addition to this users can also create alerts so that they are informed when a particular event happens.
BAM Observation Model
The BAM Observation Model is a high level view of the business process. The BAM Observation model is normally defined by a business analyst using the add-in in Microsoft Excel. The Observation Model defines the milestones and data items to collect, information about any aggregations and how they should be presented to users. The Observation Model contains the following items:
-
BAM Activities - See below for more info
-
Aggregations - An aggregation can be either real time or scheduled, see below for more info
o Aliases - There will be business terms defined for common data items to allow business users to understand what value a process may be presenting them. This is intended to ensure that developer naming conventions which may not have meaning to a business user are not displayed.
o Duration - The period of time over which the activity is monitored
o Milestone Groups - Sets of milestones which start or end the process
Aggregations
Aggregations are defined by the business analyst as pivot tables in Excel when they are creating the BAM Observation Model. The aggregation(s) define pre-calculated summaries of data which will significantly improve query response time compared to having to calculate thousands of rows of data.
Aggregations are implemented in the database as OLAP cubes.
Real-Time Aggregations
Real-time aggregations (RTA) allow a business user to see information on business processes that will change through out the day (for example if you refresh your screen the data may have changed). This is because you are seeing data based on the activity tables which contain currently executing activities. RTA data is available via dynamically created views in the BAM Primary Import Database.
This can be very useful to a business but there are some things you need to be aware of:
· Real-time aggregations significantly affect the write speed of BAM to the activity tables therefore you should only consider RTA for the most important aggregations.
Scheduled Aggregations
Scheduled Aggregations will periodically take a snap shot of the currently active activities and also the completed ones and then save this information to allow it to be queried later (For example a user may see a snapshot of this mornings order). Scheduled Aggregation data is available via dynamically created OLAP cubes in the BAM Analysis Database.
BAM Activity
An activity represents a unit of work within a business process, for example a loan application. The activity shows two key sets of things.
· Milestones - These are date based events which indicate the history of the activity such as "order completed".
· Data items - These are context type information about the unit of work such as an application number.
BAM Management Utility (BM.exe)
BM.exe is the tool which is used to deploy a BAM Observation Model to BizTalk.
Tracking Profile Editor
A tracking profile is a map between the BAM Activity and the BizTalk implementation of the activity. The Tracking Profile Editor (TPE) is a tool that allows a developer to map parts of the activity onto BizTalk artefacts such as Orchestrations or Ports.
Inteceptors
The BAM Inteceptor is used to instrument your application to capture data of interest. There are also interceptors available for WF and WCF.
Tracking Profiles Management Utility (bttdeploy.exe)
This is a command line tool used to deploy a Tracking Profile to BizTalk.
BAM Databases
There are a number of databases which BAM will install. The details of these databases can be found at the below links:
BAM Implementation Workflow
Now we are aware of some of the components of BAM we now need to know how it all fits together. This section will identify who uses what and when in the process of creating a common end-to-end BAM solution.
The following diagram shows how this fits together.
(Image courtesy of BizTalk documentation)
1. To begin with the Business analyst will use Excel to create the BAM Observation Model by using the add-in in Excel to create activities and aggregations associated with the business process.
2. BM.exe is then used to deploy the BAM Observation Model. This will do the following things:
o In the BAM Primary Import it will create the Stored Procedures, Triggers and SSIS packages
o The BAM Archive Database is defined but the objects are not created until the SSIS packages are executed;
o If it is a Scheduled Aggregation then objects are created in the BAM Star Schema database.
o If it is a Scheduled Aggregation then the appropriate OLAP cubes are created
3. The developer then uses this observation model to create the tracking profile using the Tracking Profile Editor
4. The tracking profile is deployed to the BizTalk Management Database
5. When the business process is executed the information relating to the process is now stored in the various databases depending on how the Observation Model was defined.
6. A business user can use the BAM portal or Excel to view data held in the database or in OLAP cubes to review and monitor the business process
7. If required the Business User can choose to create alerts related to data held about the business process
Alternatives
The above describes the most common way in which BAM is implemented. There are some alternatives on this which are described briefly below:
· The developer may choose to report data to BAM using the BAM API rather than creating a Tracking Profile.
· A developer may choose to use the BAM web services and then display BAM data in their own application.
· A business analyst can create visual diagram of the business process using the Orchestration Designer for Business Analysts (ODBA) which is the BAM Definition which can be imported into Excel to help complete the creation of the BAM Observation Model.
Barriers to using BAM
Although BAM is a really useful feature for BizTalk projects I think there are a few things which I would consider to be barriers which contribute to the fact that BAM is not implemented on more projects. These include: