BizTalk Server 2006

Should you cluster the BizTalk 2006 Master Secret Server?

Neil Thompson — Thu, 21 Dec 2006 14:15:00 GMT

In short, YES you should cluster the service if possible.

I have seen lots of posts on this and I think that some recent problems with SSO clustering (see my post here for fixes) may have caused people to shy away from it. If you are running SQL on a cluster, then you may as well tack on another clustered service and have that much more protection. If you are not running SQL in a cluster then clustering the Master Secret Server will probably not make much of a difference for you!

If you have a reliable cluster available and it seems to be managed well, use it and cluster the Master Secret Server. This is still the recommendation that you will get from Microsoft and still the easiest way to ensure that a hardware failure will not bring your BizTalk server to its knees. I realize that the manual process involved in bringing up another master secret server in case of a failure is simple and short, but it is still a manual process. This means that somebody has to notice the problem, diagnose the problem, and fix the problem. All of which turns into downtime in your production environment.

I have seen people take a lot of time to notice and diagnose the problem in the past, so even if the fix is a quick one, you may still have a few hours of "head scratching" time before the production support team realizes what's happening. Whereas if you have the master secret clustered, you will have a breif pause during the failover, then smooth sailing. Chances are that your apps will not be affected at all (unless you happen to be restarting your host instance at the precise instant the failover takes place, in which case you will be unable to start them properly the first time and will have to issue the restart again)

I absolutely, definitely, 100% recommend that you take the time to cluster your Master Secret Server. (Just don't enable the registry replication if you happen to get an old version of the MSS clustering docs)

SSO Master Secret Server does not start after cluster failover

Neil Thompson — Thu, 21 Dec 2006 13:30:00 GMT

This post contains instructions about modifying the registry. This is inherently risky and this post is provided AS-IS without warranty of any kind.

I recently resolved this issue for a client but I suspect that many people have it due to some incorrect documentation regarding the installation of a clustered Master Secret Server.

The Problem

SSO Master Secret server fails to start after failing over to second cluster node.

Event Logs

---------------------------------------------------------------------------------
Source: ENTSSO
EventID: 10565
Description: The secret could not be loaded from the registry. The service account for the SSO service may have been changed or the secret may be corrupted. Restore the secret from a backup file.
---------------------------------------------------------------------------------
Source: ENTSSO
EventID: 10521
Description: Could not load secrets from the registry of the master secret server.
---------------------------------------------------------------------------------

Theory 1

Perhaps the master secret was not installed on this second node. So I dig out the Master Secret from the secure file share, and run the following command in command prompt from the Enterprise Single Sign-On directory:

ssoconfig -restoresecret <keyname>
The command completes, I go to cluster administrator, take the ENTSSO service offline, bring it back online and voila! It starts fine. End of problem right? WRONG?
Just to be thorough, I fail it back over to node1 of the cluster (WHERE IT FAILS!). Now node1 exhibits the same behaviour as node2 did originally.
Then I restore they key to node1 and node2 has the same problem it had in the first place.
I scratch my head
I call MS PSS (always a good idea when you're not sure)

Pause for Thoughts

Couldn't be the key, because the node that has it restored begins to work and both nodes have it
Couldn't be the database because one node always works
Couldn't be the service account because one node always works
Each node can operate correctly so there is no defect specific to the node
Why does restoring the key on one node break the other, after all it's just a REGISTRY KEY (AHA!)

Theory 2

The registry replication configured on the ENTSSO service in the cluster definition has a domino affect and somehow messes up the registry on whatever node did not just have the key restored. If I disable that, and cleanup the registies, maybe SSO will failover properly.

Solution

Backup keys HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ENTSSO\SSOSS from both nodes (just in case)
In cluster administrator, remove registry replication for that same key on the ENTSSO "Generic Service"
Delete keys HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ENTSSO\SSOSS from both nodes (carefully)
Restart both nodes (perhaps not necessary)
Restore master secret to active node, then failover to other node
Restore master secret to this node as well.
Problem solved (test with multiple failovers to be sure)

Special thanks to Somu J. from MS PSS for the assistance on this problem.

Notepad Grows Up

Neil Thompson — Sat, 09 Dec 2006 15:23:00 GMT

Now that notepad is 20+ years old I guess Microsoft decided to beef it up a little. XML Notepad 2007 is the new kid on the block and it is a nice little tool. I will probably keep on using XmlSpy and the BizTalk Editor for schema creation, but I'll give a go at using XML Notepad to create those pesky instance documents for test submissions to my orchestrations.

It allows inplace editing of the xml document, it seems to be aware of enumeration values (always nice) and gives you a dropdown to select. It also has date inputs which have always been on my wish list. It also seems to have XSL abilities which (maybe) I could use to test biztalk maps (not the ones with scripts of course) on the XML instances that I can whip up from this little tool

You can download XML Notepad 2007 here.

Orchestrations fail to deploy due to binding errors

Neil Thompson — Thu, 12 Oct 2006 13:39:00 GMT

BizTalk 2006 seems to have a deployment quirk that has been no small source of frustration for me. I don't know if this is widespread or if I just have a bad install but it seems that whenever I change a port (add a port or remove a port) I get an error similar to the following when I try to deploy that assembly from Visual Studio,

Error 26 Failed to add resource(s). Change requests failed for some resources. BizTalkAssemblyResourceManager failed to complete end type change request. Failed to update binding information. Could not enlist orchestration 'Orchestration.UpdateRequestStatus,Orchestration, Version=1.0.0.0, Culture=neutral, PublicKeyToken=f96e91b6b75bb727'. Could not enlist orchestration 'Orchestration.UpdateRequestStatus'. All orchestration ports must be bound and the host must be set.

I've done some reading and a lot of thinking and all I can come up with is that BizTalk must be cacheing the binding information and reapplying it when I redeploy the assemblies. The part of it that burns me is that the DEPLOYMENT of the orchestrations fails because of a binding error. If you think about that for a second you realise that you can not change the binding information for assemblies that are not deployed so you get stuck.

I have not been able to find references to the binding cache, though my research is admittedly limited. On the hunch that this cache exists and is the source of my problem I tried incrementing the version on my orchestration assembly so BizTalk would ignore its previously cached info (its a new assembly version after all) and voila! It deployed my assemblies without complaining.

Two possible solutions to this that i know of are:

Change version (or key I suppose) on the orchestration assemblies
Deploy the orchestration assembly by adding it as a resource to the appropriate app in the biztalk admin tool

BizTalk 2006 FailedMessage Routing

Neil Thompson — Wed, 11 Oct 2006 11:12:00 GMT

There is no sample for this, but if I get enough requests (say five or so) I'll create one and post it.

I decided to whip up a quick prototype of FailedMessage subscription in BizTalk 2006 and I ended up hitting a couple of bumps in the road on my first shot. I'm the kind that does not stop and ask for directions, nor do I take the time to look at samples. I try to do it on my own, and failing that, I'll hit the blogs. All of that with a couple of trial and error sessions produced the following notes that I will use when trying this again:

Generating a FailedMessage

Generating a failed message is easy. Just submit a schema type that you are not looking for in any of your artifacts, or just unenlist the subscribers.
Be sure to enable FailedMessageRouting on the port that is Receiving/Sending the erroreous message

When Using A SendPort subscriber to the FailedMessage

Just create a send port and Set the filter criteria to ErrorReport.ErrorType == FailedMessage (do NOT put quotes around the failedmessage string)

When Using an Orchestration subscriber to the FailedMessage

Create a receive port that is Direct Binding (the “will be defined by filter expressions“ one)
Have your activating receive shape with the ErrorReport.ErrorType == “FailedMessage” (put quotes around the failedmessage string)
Your receive message type should just be Xmldocument

Some Insights

There are a handful of other ErrorReport fields that will help you narrow your subscription criteria if you wish I like the MessageType and ReceivePortName
If your subscription to the FailedMessages is messed up, you will see TWO reports of failed message routing per message that you submit. I can only assume that this is because the attempt to route the failedMessage also fails. If you want to explore this Disable/Enable FailedMessage routing on the port in question and see if it affects the number of reports you see.
This is all about error handling so you have to assume that some pretty significant things are allready wrong. Depending on the stage at which you get the exception that causes the failedMessage, not all of the ErrorReport feilds will be helpful. For instance, if the failure occurrs during the Xml[Dis]Assembly you can expect that the ErrorReport.MessageType will not be available.

Corollaries

This errorRouting mechanism also has the usefull behaviour of suppressing suspended messaging instances. I had a Solicit-Response call to an unreliable webservice [aren't they all], that would fail periodically. When it did fail it would suspend the orch and the message instance. When I resumed them, it would send a dup message (on from orch and one from message instance). I'm sure there's a better way to solve this one (but I don't stop for directions remember) so we enabled ErrorRouting on the send port. The consequence of this was that, on failures, only the orchestration was suspended. Now when I resume, I do not get duplicate messages and I do not have to remember to terminate the message instances.

By the way, I fiddled with the SuspendAsNonResumable on this and it never seemd to work the way that the help file described. There was a lot of sketchy behavior surrounding the resuming of a failed web call... but that can be another post.

Unable to create web reference from BizTalk project

Neil Thompson — Wed, 20 Sep 2006 13:04:00 GMT

When trying to create a web reference to an existing web service you may see one or both of the following messages

Unabled to create web reference
Failed to create web rererence

If you see these messages (or similar messages), open up the source code (if you have it) from the service that you are trying to consume. Examine the attributes of the webmethods. See if there is a “Name” attribute on the methods, and remove it if there is. After you have done this you should be able to create the web reference from the BizTalk project.

I'm not sure why this works, but it has worked for me many times. If you can't alter the attributes on the webservice, you can try using a C# class library to create the web reference for you and then call it indirectly. It's messy I know, but it works.

Using MOM 2005 to Recover ports on BizTalk 2006

Neil Thompson — Tue, 25 Jul 2006 18:40:00 GMT

This will be a semi-fictitious case study. In using MOM 2005 to recover an errant SQL port. This scenario is common and I've been asked about it a lot. This approach is the simple meat and potatoes process. There are lots of areas that could be polished up but that kind of thing can clutter up an article with too much detail.

The basic steps will be as follows:

Define a rule to detect the error condition
Define a script to correct the error condition
Wire up the script to the rule so that the rule cant trigger it.

Define a rule to detect the error condition

For whatever reason (network down, SQL down) the port was unable to commmunicate with SQL and BizTalk shut it down. This will create an event log entry similar to the following

Type:Error
EventID:5649
Source: BizTalk Server 2004/2006

The receive location "SQL://<sql name/info>" is shutting down. Details:"The error threshold has been exceeded. The receive location is shutting down."

You will setup a MOM Rule to detect this condtion. This is not a total MOM tutorial so I won't go over the details here, but the basic process is to create a new rule and use the criteria tab to specify as many details as possible from the above event log entry. Be sure the rule is using the application event log as a source.

Define a script to correct the error condition

I would normally like to use managed code, but it is nice to remove dependencies from the .NET framework version (i.e. 2.0 might not be installed) and I also like being able to issue changes without having to recompile. This is good and bad for several reasons I don't want to get into. The script that I use targets a specific port and tries to enable it. The script (written in vbscript) is the following: (it is bacially a lift from the BizTalk SDK)

'-------------------------------------------------------------------------
'
' WMI script to enable/disable the receive locations
'
'---------------------------------------------------

Option Explicit

EnableReceiveLocation "ReceivePort1", "ReceiveLocation1", "TRUE"

Sub EnableReceiveLocation(strReceivePortName, strReceiveLocationName, sEnable)

    'error handling is done by explicity checking the err object rather than using
    'the VB ON ERROR construct, so set to resume next on error.
    On Error Resume Next


    Dim objInstSet, objInst, strQuery


    'set up a WMI query to acquire a list of receive locations with the given Name and
    'ReceivePortName key values. This should be a list of zero or one Receive Locations.
    strQuery = "SELECT * FROM MSBTS_ReceiveLocation WHERE ReceivePortName =""" & strReceivePortName & """AND Name =""" & strReceiveLocationName & """"
    Set objInstSet = GetObject("Winmgmts:!root\MicrosoftBizTalkServer").ExecQuery(strQuery)

    'Check for error condition before continuing.
    If Err <> 0    Then
        PrintWMIErrorThenExit Err.Description, Err.Number
    End If

    'If Receive Location found, enable it, otherwise print error and end.
    If objInstSet.Count > 0 then
        For Each objInst in objInstSet

            'Now enable /disable receive location
            if sEnable= "TRUE" then
                objInst.Enable
            else
                objInst.Disable
            end if
            If Err <> 0    Then
                PrintWMIErrorThenExit Err.Description, Err.Number
            End If
            if sEnable = "TRUE" then
                 return 100
            else
                return 101
            end if
        Next
    Else
        WScript.Echo "No Receive Location was found matching that Name."
    End If

End Sub

'This subroutine deals with all errors using the WbemScripting object. Error descriptions
'are returned to the user by printing to the console.
Sub    PrintWMIErrorThenExit(strErrDesc, ErrNum)
    On Error Resume    Next
    Dim    objWMIError    : Set objWMIError =    CreateObject("WbemScripting.SwbemLastError")

    If ( TypeName(objWMIError) = "Empty" ) Then
        wscript.echo strErrDesc & " (HRESULT: "    & Hex(ErrNum) & ")."
    Else
        wscript.echo objWMIError.Description & "(HRESULT: "    & Hex(ErrNum) & ")."
        Set objWMIError    = nothing
    End    If

    'bail out
    wscript.quit 0
End    Sub

Sub PrintUsage()
    WScript.Echo "Usage:" + Chr(10) + Chr(10) + _
                 "cscript enabledisableallrecloc [enable/disable]" + _
                 Chr(10) + "       Example to enable all receive locations: enabledisableallreclo enable "+ Chr(10) + Chr(10)

End Sub

Wire the script to the rule

You have to go to the Response tab of the rule and use the “Execute command or batch file” option. Assuming that you have copied your vbscript file to the %PROGRAM FILES% directory on your biztalk server you can execute the following as part of your response:

Check the “Use Windows command interpreter“ (if it is appropriate for your environment)
Command line:cscript myscriptfile.vbs
Initial Directory: %PROGRAMFILES%

Common Mistakes

If your script works when executed locally but fails when triggered by MOM you probably have not set your Agent Action account . Go to the MOM adminstration console then go to Administration >> Computers >> Agent-Managed Computers. Right click on the machine in question and select “Update Agent Settings“. Choose a domain account that has the appropriate permissions (i.e. BizTalk Administrator/Operator for starting or stopping ports“
Consider the fact that this might start an infinite loop. Say SQL really is down. Then the port goes down, then MOM starts it, then it goes back down, then MOM starts it etc. Limit the retries with the Repeat count (look under advanced in the criteria tab for the rule). If you really want to get fancy, use a regular expression to define different responses based on ranges the retry count (i.e first time run script immediately, 1-5 times run script every 10 minutes, > 5 times notify operators)

BizTalk 2006 SQL Adapter problem

Neil Thompson — Fri, 21 Jul 2006 18:34:00 GMT

Breaking News - July 29 2006 - Microsoft has released a supported hotfix for this issue but you must contact microsoft support services to obtain it. The hotfix is supported but is not published on the support site because it has been classified as “confidential“. Refrence the KB article 918316 to obtain the fix

Applications that worked well under BTS 2004 may experience problems under BTS 2006 because of an apparent problem in the new SQL Adapter. The error usually happens when you are expecting multiple rows/messages back from a SQL port. The error looks something like the following.

Exception type: WrongBodyPartException
Source: Microsoft.XLANGs.BizTalk.Engine
Target Site: Void ReadMessageState(Microsoft.XLANGs.Core.Envelope, Microsoft.XLANGs.BaseTypes.XLANGMessage)
The following is a stack trace that identifies the location where the exception occured
at Microsoft.BizTalk.XLANGs.BTXEngine.BTXXlangStore.ReadMessageState(Envelope env, XLANGMessage msg)
at Microsoft.BizTalk.XLANGs.BTXEngine.BTXPortBase.ReceiveMessage(Int32 iOperation, Envelope env, XLANGMessage msg, Correlation[] initCorrelations, Context cxt, Segment s)
at CB.MMAHH_File.Orchestrations.CBHHDownloadStore.segment2(StopConditions stopOn)
at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)

There is a hotfix being developed for this issue, but it is currently (as of time of writing) in pre-release and you have to contact microsoft for it. The KB number to reference is 918316 (Which does not seem to show on the MS Site yet). The file name is 273599_ENU_i386_zip.exe.

I would just give you a link but that might do more harm than good becuase the hotfix is not fully tested. If you desperately need it and MS just can't seem to get it to you, contact me and I'll see what I can do for you.