Andy Morrison's BizTalk Blog

Service Virtualization with the Microsoft Managed Services Engine

Andy Morrison — Fri, 22 May 2009 20:49:03 GMT

I presented at the Twin Cities Connected Systems User Group last night on the Managed Services Engine. You can get the presentation here.

The Managed Services Engine is a CTP technology that acts as a service intermediary. It helps abstract consumers form the services that they call, performing things such as xslt mapping, protocol switching, version mapping and service stubbing (useful for testing unhappy service respone paths as well as development of the consumer before the service is really ready.)

You can get the Managed Services Engine on CodePlex: http://www.codeplex.com/servicesengine

BAM Archiving/BAM_DM_ observations

Andy Morrison — Thu, 21 May 2009 14:52:01 GMT

From the BizTalk documentation: “The DTS package, BAM_DM_<ActivityName>, performs the partitioning and archiving/purging. Each time this package runs, it truncates another partition and archives/drops all partitions that are outside the online window.”

Here’s my restatement of this: The BAM_DM_<ActivityName> package (SSIS or DTS depending on your version of BizTalk and despite the fact that the docs refer to DTS only) performs two operations: building partition tables and archiving those partitions to the BAMArchive database when a partition’s creation date is past the online window.

If you open up a BAM_DM_<ActivityName> SSIS/DTS package and walk through it you’ll see that the second step calls the bam_Metadata_SpawnPartition stored procedure. Toward the end of that sproc it inserts a new record into the bam_Metadata_Partitions table setting the CreationTime’s value but not the ArchivedTime value which means it will be NULL.

A later step in the package calls the bam_Metadata_BeginArchiving stored procedure. Toward the end of this stored procedure it performs an SQL Update on the bam_Metadata_Partitions table where the ArchivedTime is null and the CreationTime is less than the online threshold (derived from the online window.) If the Update succeeds any partitions that should be archived have their ArchivingInProgress value set to 1 (this is set so that if the package runs again, before it completes, the second run will essentially just exit without doing anything.)

Later steps (two of them) in the package perform the actual archiving; they examine the CreationTime and ArchivedTime to figure out what they should archive, using the same logic outlined above (for the . bam_Metadata_BeginArchiving sproc).

There are many more details but hopefully provides enough detail for the takeway.

Takeaway: your BAM_DM_<ActivityName> package almost certainly needs to run twice to actually archive data to the BAMArchive database because the time between when the partition is created during the second step of the package and when archiving will happen (later steps) isn’t going to be greater than your online window. The lowest you can set your online windows is 1 minute. Unless you have massive quantities of BAM data, and a slow SQL Server, it isn’t going to take more than a minute between when the bam_Metadata_SpawnPartition sproc runs (which sets the CreationTime) and the archiving steps execute. So, the first time you run your package it will create the partitions; the second time it will archive those partitions, if they fall outside the online window. This means you need to think carefully about how often you schedule your BAM_DM_<ActivityName> packages and how that is related to the online window.

For additional context reference the Business Activity Monitoring in Depth for Developers whitepaper here: http://download.microsoft.com/download/D/D/A/DDA95D1F-14D4-49A2-B2C8-E244535E8502/BAM_for_Developers.docx

The BAM deployment failed. Encountered error while executing command on SQL Server. String or binary data would be truncated.

Andy Morrison — Mon, 04 May 2009 16:45:41 GMT

I ran into an issue with bm.exe this morning.

Here’s the error I saw when running the bm.exe depoy-all command on a spreadsheet.

Deploying View... ERROR: The BAM deployment failed.

Encountered error while executing command on SQL Server "MyMachineName".

String or binary data would be truncated.

The statement has been terminated.

I ran SQL Profiler the next time I executed this (as well as adding the –Trace:On parameter to the deploy-all command which _never_ seems to add any additional information. After a little bit of sleuthing it looked like this was the last statement executed before profiler showed sql statements that were deleting BAM tables, sprocs, etc.

-- Register activity in RTA metadata table

INSERT [dbo].[bam_Metadata_RealTimeAggregations]

(

CubeName,

RtaName,

RTAWindow,

Timeslice,

ConnectionString

)

VALUES

(

N'CreateOESIView',

N'CreateOESIPT',

3600, -- RTA window defaults to 60 hours

60, -- Time slice defaults to 60 mins

N'[HugeStringHere]’

The ConnectionString column’s max size is 3600 and you guessed it, my [HugeStringHere] value was larger than that. The string was actually not just a connection string and included CREATE CUBE and other statements in it.

It turns out that the PivotTable that I created in Excel for this particular view was too big – as in it had too much stuff in it (Dimensions, Measures). I created a smaller pivot table with fewer Dimensions in it and then the activity deployed without problems.

BizTalk vs. WF/WCF Comparison (Smack Down) Article published

Andy Morrison — Thu, 19 Jul 2007 16:32:33 GMT

In the BizTalk Hotrod quarterly ezine. You can find a condensed version of the article at http://www.biztalkhotrod.com

The full version (huge!) can be found at http://biztalkhotrod.com/Documents/BizTalk%20Smack%20Down%20Full.pdf

The article was a ton of work but was well worth the effort. The intent was to have some fun comparing BizTalk and WF/WCF. Let me know if you have any feedback!

Twin Cities Spring 2007 Code Camp - Developing Business Activity Monitoring Solutions with BizTalk Server’s BAM Features

Andy Morrison — Sun, 06 May 2007 19:57:38 GMT

Thanks to all who attended my session on BAM last weekend.

You can find the presentation and demo source code here. Note that I did not get the WCF Interception running so the XML file for that is not included - I'll discuss that in a follow up post in the next week or two.

To run the demo artifacts you'll need BizTalk Server 2006 R2 Beta 2, .NET 3.0 installed and Excel 2007 installed.

BizTalk Hotrod Magazine

Andy Morrison — Mon, 23 Apr 2007 19:39:42 GMT

I'm a little late in posting about this but there is a new BizTalk-specific "magazine" called BizTalk Hotrod: http://biztalkhotrod.com/default.aspx

The first issue was published just a few days ago.

Enjoy!

Twin Cities Code Camp - Spring 2007

Andy Morrison — Mon, 23 Apr 2007 19:37:14 GMT

The second Twin Cities Code Camp will occur this Saturday, April 28th. Here's the website, which includes a schedule and list of sessions: http://www.twincitiescodecamp.com/TCCC/Default.aspx

I'll be presenting on utilizing BizTalk's BAM features from non-BizTalk applications.

Repost - BizTalkRocks 'Debugging Tips and Tricks' Webcast

Andy Morrison — Wed, 06 Dec 2006 13:29:00 GMT

This post disappeared from a previous blog of mine so I'm reposting it.

---

'Thank you' to everyone who attended the Debugging Tips and Tricks webcast that I presented today for the BizTalkRocks community. You can find the presentation at http://www.biztalknuggets.com/presentations/BizTalk Debugging Tips and Tricks.zip

I recorded the webcast. I will repost this article and include a link to the recorded webcast when it is posted.

If you have any questions on the presentation or the webcast please contact me through the Contact link on this blog.

Update (12/9/05): I think I skipped a slide yesterday. If you download the presentation, search for 'BizUnit' to find another 'Other Tools...' slide. Sorry about that.

Update (3/26/07): Updated the link to the presentation.

Changing PivotTable Names for the BAM Portal (Before Deploying the BAM Definition)

Andy Morrison — Tue, 05 Dec 2006 13:47:00 GMT

If you’ve worked with BAM and the BAM Portal in BizTalk 2006 you’ve noticed that the PivotTables in the My Views - Aggegations section shows up with ugly names like PivotTable13, PivotTable14, etc. This is fine when you are doing POCs with a small number of Views/PivotTables but it can quickly become confusing – and more importantly these names aren’t appropriate when you’re deploying a real BAM solution to end users.

If you want to change these names before you deploy your BAM Definition you need to:

Export the BAM Definition to XML using the BAM menu in Excel.
Open the exported BAM Definition
Update each /BAMDefinition/Extension/OWC/PivotTableView/PivotTable/PivotView/Label/Caption element’s text to use a more appropriate name; note that this applies to both Real-Time aggregations and scheduled aggregations.

[Update 12/7/06 - You can also right click the PivotTable in Excel, select Table Options and then change the Name.]

Now when deploying the BAM Definition, use the .xml version of it and then you will see your version of the PivotTable names in the My Views section of the BAM Portal.

Twin Cities User Group Website

Andy Morrison — Wed, 06 Dec 2006 13:32:00 GMT

The Twin Cities BizTalk User Group now has a website, you can find it here: http://www.mnbiztalk.com/

If you are a BizTalker, and live in or around the Twin Cities, and you haven't attended a session yet - I strongly encourage you to attend. The sessions are great and the environment isn't as "stiff" as some of the other users groups out there :)

BizTalk Article Published in August .NET Developers Journal

Andy Morrison — Wed, 06 Dec 2006 13:39:00 GMT

It took a while for the online version to appear but you can find the article, "Introduction to BizTalk Server's Features and Benefits" , here.

Repost - IIS 6.0, Integrated Windows Security, Application Pools with Domain Accounts and ‘The target principal name is incorrect’

Andy Morrison — Wed, 06 Dec 2006 13:27:00 GMT

This post disappeared from a previous blog of mine so I'm reposting it.

---

A couple of weeks ago I was having difficulty turning on Windows Integrated security for a couple of virtual directories that were interfaces to the SOAP and HTTP adapter. I initially had Anonymous and Integrated Windows turned on. As soon as I turned off anonymous and left only Integrated Windows security turned on I would get prompted to enter my credentials. After three failed attempts to login I would get the following error message: ‘HTTP Error 401.2 - Unauthorized: Access is denied due to server configuration.’

I assumed that I didn’t have the ACLs properly configured on the directories and files that the virtual directories managed but after reviewing their configuration everything checked out.

I stumbled across a reference to AuthDiag, a new tool from MS that helps you troubleshoot authentication and authorization issues with IIS 5.x and 6.0. I installed the tool, ran the ‘Check Authentication’ test on the virtual directories and quickly discovered that my issues had something to do with the fact the identity for the application pools that my virtual directories were running in wasn’t configured properly. Auth Diag kept giving me a failure with the message ‘Service principal name (SPN) for user xx\yy not found in Active Directory’ where xx was the domain and yy was the user name for the account I was using as the application pool identities. (As you probably know, if you have multiple BizTalk Servers in your BizTalk Group then you must use domain accounts for you host instance credentials and I was using the same account for the application pool identities.)

After a suggestion from a colleague, I switched the identities for the application pools to a local account (Network Service) and the prompt and 401 immediately went away. Of course the requests couldn’t make it into BizTalk because the local Network Service Account didn’t have the permissions to reach the BizTalk MessageBox database.

Another colleague suggested using wfetch.exe, part of the IIS 6.0 Resource Kit to help debug my issue. I ran wfetch.exe using NTLM (for the ‘Auth’ value in the ‘Authentication’ group box) and could access the virtual directories without problems. However, when using either Kerberos or Negotiate I saw the following error message in the response from the IIS: ‘0x80090322 (The target principal name is incorrect.): Unable to InitializeSecurityContext’. This article describes the differences between these three options although it describes these in the context of the IIS authentication and not wfectch.exe

I was stumped until I found the Configuration Application Pool Identity in the IIS 6.0 documentation. According to that page, there are issues with using Kerberos (which may be used when you use Integrated Windows Security.) The solution to my problem was that I needed to run the setspn.exe tool in order to set a Service Principal Name (SPN) on the account configured as the Identity for the application pools. We ran setspn.exe using the machine name and the machine’s fully qualified domain name, and the issue immediately went away:

setspn -A HTTP/servername domain\account

setspn -A HTTP/servername.fullyqualifieddomainname domain\account

You can find the setspn.exe tool on the Windows 2003 CD-ROM or .ISO at \support\tools\suptools.msi

One thing to note about setspn.exe is that you must be a domain administrator, of the domain that the account is in, that you use as a parameter to setnspn.exe (which is also the account you are using for the application pool identity. There is a note here (click on the ‘Troubleshooting Setspn’ link) that says the permission necessary for running the setspn.exe can be delegated.

In summary, you must execute setspn.exe if you are using domain accounts for your application pool identities if you are using Integrated Windows Security.

Repost - BizTalk 2004 Developer Workstation Installation Tips

Andy Morrison — Wed, 06 Dec 2006 13:26:00 GMT

This post disappeared from a previous blog of mine so I'm reposting it.

---

I've installed BizTalk 2004 on a developer workstation several times at this point. For a developer workstation I recommend the following:

1) If you do 'regular' ASP.NET development on your workstation and you will continue to do so after installing/configuring BizTalk then when you are configuring Windows SharePoint Services (WSS) do not configure it to use your default website. Create a second website in IIS and point to this site during WSS configuration (and later during BizTalk configuration point to this site instead of your default web site for the BAS site location.) If you don't do this you will have to put your ASP.NET applications on a second website that doesn't run on port 80 or you will have to configure WSS to not 'apply' to each Virtual Directory that you want to use for ASP.NET applications.

2) I have never succesfully been able to create the WSS configuration db from the WSS configuration wizard. I always get an access denied error when it tries to create the configuration database, complaining about NT Authority\Network Service not having appropriate access to the database (or something like that.) Instead, I create the WSS configuration database prior to configuring WSS using the Using the Command Line to Create or Connect to a Configuration Database section of the Administrator's Guide for SharePoint Services. This section describes how you can use the following command to create the config database:

stsadm -o setconfigdb -ds -dn

You can then point to the database you created with stsadm during the WSS configuration.

Repost - Reusing the BizTalk Server 2004 Schema Generator Wizard's Library

Andy Morrison — Wed, 06 Dec 2006 13:25:00 GMT

This post disappeared from a previous blog of mine so I'm reposting it.

---

I've been doing some research on custom adapter development and one of the issues that came up was figuring out how to do the schema generation. BizTalk Server 2004 includes a wizard that will generate schemas from Xml instance documents. I decided to figure out how it worked - and if it could be reused.

It is possible to reuse this functionality. Here's how you do it:

Create a console project.

Reference the Microsoft.BizTalk.WFXToXSDGenerator.dll and XSDInfer.dll assemblies in the [installDir]\Microsoft BizTalk Server 2004\SDK\Utilities\Schema Generator directory.

Reference the Microsoft.BizTalk.SchemaEditor.Extensibility.dll assembly in the [installDir]\Program Files\Microsoft BizTalk Server 2004\Developer Tools directory.

Pull in the Microsoft.BizTalk.WfxToXSDGenerator namespace.

Add the following code to your Main method:

WfxToXSDGenerator gen = new WfxToXSDGenerator();

gen.GenerateSchema(args[0], args[1]);

object[] warnings = gen.Warnings();

object[] errors = gen.Errors();

object[] schemas = gen.ReferencedSchemas();

if(warnings.Length != 0)

{

Console.WriteLine("Warnings were found:");

for(int i = 0; i < warnings.Length; i++)

{

Console.WriteLine(warnings[i].ToString());

}

Console.ReadLine();

return;

}

if(errors.Length != 0)

{

Console.WriteLine("Errors were found:");

for(int i = 0; i < warnings.Length; i++)

{

Console.WriteLine(errors[i].ToString());

}

Console.ReadLine();

return;

}

if(schemas.Length !=0)

{

Console.WriteLine("The following schemas were generated:");

for(int i = 0; i < schemas.Length; i++)

{

Console.WriteLine(schemas[i].ToString());

}

Console.ReadLine();

return;

}

Notes:

The first argument to the GenerateSchema method is the location of the Xml instance document. The second argument is the output directory for the generated schemas.
A Warning and Errors array is present; oddly you access these through methods, not properties. I didn't run into any errors or warnings so I didn't investigate their Types; my assumption is that they are string but it would be easy for you to figure this out if you run into issues.
GenerateSchema() doesn't overwrite schemas you already created with it; it adds a n (a number starting with 1) to the end of the filename (i.e. sample1.xml)
There does appear to be another MS Xsd inference assembly out there (http://www.fawcette.com/xmlmag/2002_11/online/xml_rjennings_11_11_02/); perhaps the assembly is later version of this assembly.Should be fairly robust given this is what the Wizard uses (inferred from the directories.)
I did not investigate the DTD version of the generator.

Why would you use reuse this library? A couple of thoughts come to mind: when automating batch schema generation and when writing custom application adapters. Obviously, there are other resources available on the web (and within VS.NET) that can generate schemas from Xml instance documents but I assume that this library has been put through a rigorous QA process (although it is a sample according to this post http://groups-beta.google.com/group/microsoft.public.biztalk.sdk/browse_thread/thread/913c7fa57fc127a/2ac3418b674f5da8?q=ISchemaGenerator#2ac3418b674f5da8 [I wonder why it is in the utilities directory if it is a sample?]) given its usage in BizTalk Server 2004.

Repost - Preparing to Take Your BizTalk Server 2004 Application to Production

Andy Morrison — Wed, 06 Dec 2006 13:24:00 GMT

This post disappeared from a previous blog of mine so I'm reposting it.

---

Are you read to take your BizTalk Server 2004 application to production?

Integration applications are usually very complex; they have a lot of moving pieces, they use many different technologies and they are often built to automate processes that are not understand nearly as well as the business units involved think they are understood. In addition to the technical problems related to integration applications, integration problems have a corresponding set of issues centered around communication difficulties. Often these communication difficulties arise because the different teams involved in the project do not understand ant technologies other than their own, because people of various technical levels are involved in project and because project teams do not reside at the same physical location.

Because of these complexities it is easy to overlook items that should be handled prior to your BizTalk Server 2004 application going live. The article contains suggestions for items you should resolve prior to going live.

Deployment

Is your deployment automated? Fully automated?
Have you defined your rollback procedures?

Application Architecture Review

Have you covered all of your error handling scenarios from you requirements?
Will HAT provide all of the necessary debugging, process analysis and replay requirements for your application? If not you may need to
Database recovery, backups, failover, etc.
Logging - where, how, when?
How are you going to monitor you application and environment? What happens after hours? Have you considered Microsoft Operations Manager?
How are you going to implement notification? Who gets notified? When? Will you filter so that you don't send out notifications for each and every BizTalk related event log entry?
Will you have enough information to debug your problems (they will occur!)?
Can you really guarantee delivery? Did you design your orchestrations correctly?
Have you reviewed the Performance and High-availability whitepapers?

Hardware Architecture Review

Are your environments the same from Development to Production?
If necessary, have you really eliminated single-points of failure? What about the resources BizTalk will depend on such as MQSeries server, databases, etc. You are as weak as you weakest point.
Can you really meet you availability requirements?
If necessary, do you have an environment that you can perform load and performance testing in? Do you have access to it? You will need access to it so that you can change tuning settings.

Operations Staff

Do you know how to work with HAT?
Will you have access to HAT in you production environment? If not, how will you work with your operations staff during issues?
Have you trained your operations staff on HAT?
Does your operations staff know how to use your deployment tool?
Does your operations staff know how to use BizTalk Administration?
Have you planned your situation handling procedures and documented them?
Does operations understand all of your application scenarios?
Does operations understand the service-level agreement for you application? For your platform?

Testing

Have you tested failover scenarios?
Can you really run your business scenarios in your pre-production environment? If not, be prepared for issues in production soon after you deploy there.
Are your environments the same from Development to Production?
For applications that will be under heavy load, when you PoC'ed did you test the PoC under a full load? You might be surprised at what happens if you only tested with a percentage of your full load and extrapolated.
For applications that will be under heavy load, have you tested all of the adapters separately? They behave very differently under load and have different tuning settings.
Have you tested all of you unhappy paths?
Does everyone know what end-to-end testing means? Are you sure?
Do you have testing schedule defined? This can be very difficult to plan given that integration applications often have many systems and departments involved.

Business

Have you reminded the business units about the new process changes?
Have you defined the communication paths between the involved business unit/parties when problems occur? This is also something that your operations staff will have to know about.
Does the business know how/when they will be notified of issues; this includes automated notifications and SLA requirements for how the operations staff will notify customers about issue resolution?

This list of items is not exhaustive and there are likely many other things that you should consider for your particular situation prior to taking your BizTalk Server 2004 application to production; however, I have provided a sample of items that you may have not thought of and that will hopefully provide some motivation to discover other items for you particular situation.

If you have other suggestions for this list, let me know.