Kerberos Multi-hop Delegation Troubleshooting Tale

One of our .net application teams has had a problem for quite a while that related to impersonation and kerberos multi-hop delegation which had proven quite difficult to resolve. We eventually resolved this and I thought it would be worth popping a little bit of information about it out there incase anyone else has similar problems.

We had two web services with 2 methods which participate in a Kerberos multi-hop delegation scenario using WSE 2. One of the methods works fine all of the time and the other method intermittently was having problems.

Symptoms

The symptoms we experienced were:

• For method 1 everything worked fine all of the time
• Everything appeared to be working fine for method 2 but after a period of time it would start getting errors which indicated the Kerberos token had expired
• Method 2 also looked occasionally like the wrong user identity had been flown over the wire to the back end service
• The calls to method 2 which stopped working after a period of time seemed to coincide with the expiry of the Kerberos ticket (around a week to 10 days can't remember the exact time frame but it coincided with the cache duration of a Kerberos ticket)
• After an IIS reset things worked fine straight away for a period of time and then would stop working after the time period above
• The error message we were getting was: The Kerberos credential handle could not be acquired. The AcquireCredentialsHandle call returned the following error code: A specified logon session does not exist. It may already have been terminated.

Troubleshooting

This was very painful to troubleshoot, the .net team were struggling to recreate this problem in any of their test environments. Through the addition of some instrumentation they were able to derive what was going on.

What was happening

When the development team originally coded web service A for method 2 they had implemented a singleton or caching pattern of the proxy object for web service 2. This meant they did not need to construct a new instance each time.

This meant that when the method 2 was called then the WSE policy was applied and the Kerberos token used etc but this object must internally be handling things like WS-Conversation and some kind of caching so that the Kerberos token can be reused over the lifetime of that proxy instance. In essence the first user to create the object's identity was used for every subsequent usage of this object when applying a Kerberos token to the web service calls made with this proxy.

Also this would explain why after a period of time the calls fail because the Kerberos ticket obtained for that user would have expired.

Solution

The solution was to change the code in web service A so that the proxy to web service B was not cached or a singleton and so that a new instance of this proxy was created each time.

Would this apply to WCF

At this stage I have not tested if WCF would get the same problem but I do know you run into issues in various configurations if you haven't disposed of a proxy class correctly so generally it has become a very common practice for developers to safely dispose of their WCF proxies so this tends to be less of an issue. In WSE 2 or 3 developers are not used to having to use a dispose pattern around web service proxies

Lessons to learn

Some take away's for this include:

• It is a lot more likely to be your own code than framework code causing obscure problems like this
• Impersonation can be a dangerous thing so ensure you test and code review any usage of it

Credit goes to Matt Buckley who found the needle in the stack of needles in the custom .net application and found the cached proxy instance through extensive analysis of the custom code.

• Share This Post:
• 
• 
• 
•   Short Url: http://wblo.gs/cOI

Posted On Friday, November 04, 2011 10:46 AM | Feedback (0) |

BDD and BizUnit 2 – Not as cool as combining BizUnit 4 & SpecFlow but still works ok

I've been back working with BizTalk 2006 R2 for a customer recently and I've become such a fan of the BDD style acceptance tests I've done in the past with BizTalk 2010 that its quite frustrating working back in Visual Studio 2005 and not being able to use Specflow alongside BizUnit 4 like I described in the recent videos on these subjects

• BizTalk & BDD Part 1
• BizTalk & BDD Part 2

In BizTalk 2006 development your back to the older style xml bizunit tests and we were looking at some old tests written a long time ago and frankly it's hard to remember what they were supposed to do. This is where the separation between intention and implementation of the test is so important. Since there isn't a great deal we can do about this in terms of upgrading the technology we took the very simple approach of annotating the BizUnit tests using the Gherkin style so that before each test step or group of test steps there was something to explain what was intended to happen. The other benefit is it really encourages the TDD approach.

In the first example below you can see when writing a new test we have simply used the Gherkin syntax Given/When/Then to record the requirements and definition of what needs to happen in this test.

<TestCase testName="VAU.CardChanges">

<TestSetup>

<!-- GIVEN: The test event queue is clean-->

<!-- GIVEN: The test folder is empty - Acquirer folders -->

<!-- GIVEN: The test folder is empty - CardSystem folders -->

<!-- GIVEN: The test folder is empty - CRM folders -->

<!-- GIVEN: The acquirer file generation numbers are zero -->

</TestSetup>

<TestExecution>

<!-- WHEN: TWS calls BizTalk to trigger the validation of payments -->

<!-- THEN: The response will indicate that validation has started -->

<!-- THEN: BizTalk will start the batch processing orchestration -->

<!-- THEN: BizTalk will send the batch to the acquirer -->

<!-- THEN: The acquirer will find the batch in their input folder -->

<!-- THEN: The acquirer will put the response in their output folder -->

<!-- THEN: BizTalk will recieve an acknowledgement from the acquirer -->

<!-- THEN: BizTalk will recieve a response file from the acquirer -->

<!-- THEN: BizTalk will process any card updates -->

<!-- THEN: The card will be updated in CARDSYSTEM-->

<!-- THEN: The validation batch will be updated in CARDSYSTEM-->

<!-- THEN: The card will be updated in CRM-->

<!-- THEN: Biztalk will have completed processing the validation batch response -->

<!-- THEN: The card update received by CRM will be valid -->

<!-- THEN: The card update received by CARDSYSTEM will be valid -->

</TestExecution>

<TestCleanup>

<!—Clean Test Folders -->

</TestCleanup>

</TestCase>

The fully completed and implemented test looks like the below example. Now hopefully you can clearly see how important this simple documentation around the tests are for people who may look at this in the future.

<TestCase testName="VAU.CardChanges">

<TestSetup>

<!-- GIVEN: The test event queue is clean-->

<TestStep

assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"

typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.CleanEvents">

</TestStep>

<!-- GIVEN: The test folder is empty - Acquirer folders -->

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">

<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</Directory>

<SearchPattern>*.*</SearchPattern>

</TestStep>

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">

<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerResponse</Directory>

<SearchPattern>*.*</SearchPattern>

</TestStep>

<!-- GIVEN: The test folder is empty - CardSystem folders -->

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">

<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CardSystemWS</Directory>

<SearchPattern>*.*</SearchPattern>

</TestStep>

<!-- GIVEN: The test folder is empty - CRM folders -->

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">

<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CRMWS</Directory>

<SearchPattern>*.*</SearchPattern>

</TestStep>

<!-- GIVEN: The acquirer file generation numbers are zero -->

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.DBExecuteNonQueryStep">

<DelayBeforeExecution>1</DelayBeforeExecution>

<ConnectionString>Persist Security Info=False;Integrated Security=SSPI;Server=localhost;Database=PaymentCardServices</ConnectionString>

<SQLQuery>

<RawSQLQuery>UPDATE FileNumber SET FileNumber = 1 WHERE ID = 'NextFGN'</RawSQLQuery>

</SQLQuery>

</TestStep>

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.DBExecuteNonQueryStep">

<DelayBeforeExecution>1</DelayBeforeExecution>

<ConnectionString>Persist Security Info=False;Integrated Security=SSPI;Server=localhost;Database=PaymentCardServices</ConnectionString>

<SQLQuery>

<RawSQLQuery>UPDATE FileNumber SET FileNumber = 1 WHERE ID = 'ActiveFGN'</RawSQLQuery>

</SQLQuery>

</TestStep>

</TestSetup>

<TestExecution>

<!-- WHEN: TWS calls BizTalk to trigger the validation of payments -->

<TestStep assemblyPath="Acme.BizTalk.Fx.Testing.dll" typeName="Acme.BizTalk.Fx.Testing.BizUnit.Wse2WebServicesClientProtocolStep">

<WebServiceUrl>http://localhost/Acme.CardProcessing.Secure/CardService.ashx</WebServiceUrl>

<ProxyTypeName>Acme.CardProcessing.Utilities.Testing.TestProxies CardService, Acme.CardProcessing.Utilities</ProxyTypeName>

<ProxyMethodName>SubmitBatchValidation</ProxyMethodName>

<ExpectingError>false</ExpectingError>

<InputMessageTypeName>Acme.CardProcessing.Utilities.Testing.TestProxies.CardValidationBatch, Acme.CardProcessing.Utilities</InputMessageTypeName>

<MessagePayloadPath>..\..\..\Acme.CardProcessing.AcceptanceTests\Features\BatchValidation\TestData\CardChanges.Input.xml</MessagePayloadPath>

<!-- THEN: The response will indicate that validation has started -->

<ValidationStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.XmlValidationStep">

<XPathList>

<XPathValidation query="/ValidationBatchesReturn/Return">0</XPathValidation>

</XPathList>

</ValidationStep>

</TestStep>

<!-- THEN: BizTalk will start the batch processing orchestration -->

<TestStep assemblyPath="Acme.CardProcessing.AcceptanceTests.dll" typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">

<Action>Batch validation started</Action>

<RemoveEventFromQueue>true</RemoveEventFromQueue>

<SecondsToWait>30</SecondsToWait>

</TestStep>

<!-- THEN: BizTalk will send the batch to the acquirer -->

<TestStep

assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"

typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">

<Action>Batch sent to acquirer</Action>

<RemoveEventFromQueue>true</RemoveEventFromQueue>

<SecondsToWait>30</SecondsToWait>

</TestStep>

<!-- THEN: The acquirer will find the batch in their input folder -->

<TestStep assemblyPath="Acme.BizTalk.Fx.Testing.dll" typeName="Acme.BizTalk.Fx.Testing.BizUnit.FileExistsStep">

<Timeout>60</Timeout>

<DirectoryPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</DirectoryPath>

<SearchPattern>*.*</SearchPattern>

<ExpectedNoOfFiles>1</ExpectedNoOfFiles>

</TestStep>

<!-- THEN: The acquirer will put the response in their output folder -->

<TestStep assemblyPath="Acme.CardProcessing.AcceptanceTests.dll" typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.ProcessAcquirerFileStep">

<DirectoryPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</DirectoryPath>

<SearchPattern>INZET.DANG.MDNDMDM</SearchPattern>

<ResponsesXmlFileName>..\..\..\Acme.CardProcessing.AcceptanceTests\GenericTestData\HSBCDefaultBatchResponse.xml</ResponsesXmlFileName>

<SchemaFileName>..\..\..\Acme.CardProcessing.AcceptanceTests\GenericTestData\BatchResponses.xsd</SchemaFileName>

<CreationPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerResponse</CreationPath>

<InvalidResponse></InvalidResponse>

<MultipleResponse>false</MultipleResponse>

</TestStep>

<!-- THEN: BizTalk will recieve an acknowledgement from the acquirer -->

<TestStep

assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"

typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">

<Action>Acknowledgement received from acquirer</Action>

<RemoveEventFromQueue>true</RemoveEventFromQueue>

<SecondsToWait>30</SecondsToWait>

</TestStep>

<!-- THEN: BizTalk will recieve a response file from the acquirer -->

<TestStep

assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"

typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">

<Action>Acquirer response received</Action>

<RemoveEventFromQueue>true</RemoveEventFromQueue>

<SecondsToWait>30</SecondsToWait>

</TestStep>

<!-- THEN: BizTalk will process any card updates -->

<TestStep

assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"

typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">

<Action>Processing card updates</Action>

<RemoveEventFromQueue>true</RemoveEventFromQueue>

<SecondsToWait>30</SecondsToWait>

</TestStep>

<!-- THEN: The card will be updated in CARDSYSTEM-->

<TestStep

assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"

typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">

<Action>CardServicesStub.CardServices.UpdatePaymentCards Called</Action>

<RemoveEventFromQueue>true</RemoveEventFromQueue>

<SecondsToWait>30</SecondsToWait>

</TestStep>

<!-- THEN: The validation batch will be updated in CARDSYSTEM-->

<TestStep

assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"

typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">

<Action>CardServicesStub.CardServices.UpdateValidationBatch Called</Action>

<RemoveEventFromQueue>true</RemoveEventFromQueue>

<SecondsToWait>30</SecondsToWait>

</TestStep>

<!-- THEN: The card will be updated in CRM-->

<TestStep

assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"

typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">

<Action>CRMApplicationStub.FinanceRefService.UpdateCreditCardDetails Called</Action>

<RemoveEventFromQueue>true</RemoveEventFromQueue>

<SecondsToWait>30</SecondsToWait>

</TestStep>

<!-- THEN: Biztalk will have completed processing the validation batch response -->

<TestStep

assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"

typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">

<Action>Batch validation completed</Action>

<RemoveEventFromQueue>true</RemoveEventFromQueue>

<SecondsToWait>30</SecondsToWait>

</TestStep>

<!-- THEN: The card update received by CRM will be valid -->

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileValidateStep">

<Timeout>6000</Timeout>

<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CRMWS</Directory>

<SearchPattern>*.*</SearchPattern>

<DeleteFile>false</DeleteFile>

<ValidationStep assemblyPath="Acme.CardProcessing.Utilities.dll" typeName="Acme.CardProcessing.Utilities.Testing.BizUnit.XmlComparisonValidationStep">

<CompareTo>CardChanges.CRM.UpdateCards.xml</CompareTo>

</ValidationStep>

</TestStep>

<!-- THEN: The card update received by CARDSYSTEM will be valid -->

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileMultiValidateStep">

<Timeout>6000</Timeout>

<DirectoryPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CardSystemWS</DirectoryPath>

<SearchPattern>*.*</SearchPattern>

<ValidationStep assemblyPath="Acme.CardProcessing.Utilities.dll" typeName="Acme.CardProcessing.Utilities.Testing.BizUnit.XmlComparisonValidationStep">

<CompareTo>CardChanges.CardSystem.UpdateCards.xml</CompareTo>

</ValidationStep>

<ValidationStep assemblyPath="Acme.CardProcessing.Utilities.dll" typeName="Acme.CardProcessing.Utilities.Testing.BizUnit.XmlComparisonValidationStep">

<CompareTo>CardChanges.UpdateBatch.xml</CompareTo>

</ValidationStep>

</TestStep>

</TestExecution>

<!-- Clear out all test folders -->

<TestCleanup>

<!-- Acquirer folders -->

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">

<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</Directory>

<SearchPattern>*.*</SearchPattern>

</TestStep>

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">

<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerResponse</Directory>

<SearchPattern>*.*</SearchPattern>

</TestStep>

<!—Card App folders -->

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">

<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CardSystemWS</Directory>

<SearchPattern>*.*</SearchPattern>

</TestStep>

<!-- CRM folders -->

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">

<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CRMWS</Directory>

<SearchPattern>*.*</SearchPattern>

</TestStep>

</TestCleanup>

</TestCase>

• Share This Post:
• 
• 
• 
•   Short Url: http://wblo.gs/cOH

Posted On Friday, November 04, 2011 10:40 AM | Feedback (0) |

Labelling a Build in TFS from Cruise Control

This is just a reminder for my self of how and why we do this.

We have 2 projects within a TFS project collection for our integration component developments.  We have:

1. A project for .net based integration projects

2. A project for BizTalk based integration projects

The main reason we do this is so we dont have loads of TFS projects as we have a significant number of components but also we want some different rules around check in and source control locks etc etc.....

At this stage our build servers are still running cruise control rather than Team Build so one of the problems i was finding was when cruise control labelled a build in TFS i couldnt tell easily in the label search which component or folder the build label related to.  We modified the build scripts to do a custom label with TF.exe so that we could put a specific label from each build.  We used the following command:

<Exec Command='"C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\tf" label "$(CCNetProject).$(CCNetLabel)" $(MsBuildProjectDirectory) /recursive'/"/>

Using this command means that the build label is applied at the right place for the workspace but also the label includes the cruise control project name making it easy to workout what the label was actually for rather than just being a bunch of version numbers.

There is probably a better way to do this but its working fine for us.

• Share This Post:
• 
• 
• 
•   Short Url: http://wblo.gs/cOF

Posted On Friday, November 04, 2011 10:11 AM | Feedback (1) |