Friday, February 03, 2012
Before I get into this I am not an Office Live Small Business or Office 365 expert but I have used Office Live Small Business for number of years as a light weight way of managing some parts of my business. I now need to migrate to Office 365 and one of the areas which concerned me was around custom lists which had attachments. I had a number of these with lots of rows and I had been waiting for some information on how these would be migrated and hoped they would just be migrated for me automatically.
When the migration guidance came out this was unfortunately not the case so I thought this post may help some others who have the same task to perform over the coming months before the final closure of OLSB.
Step 1: Replicate the list structure
In your Office Live Small Business account open up the custom list and go to its settings so you can see the structure of the list.
In your Office 365 account create the new custom list to replicate the structure from Office Live Small Business.
A couple of recommendations here:
- You may want to consider leaving the constraints on columns such as choices etc for later and initially create the columns with simple types then fix them later
- Ensure the columns are in the same order to make things simpler
Step 2: Open Your existing list in Microsoft Access
In your Office Live Small Business custom list use the Actions menu and select Open with Access.
This will open the list in Microsoft Access so you will obviously need it installed.
When the list opens in access choose the option to export a copy of the data and an appropriate location to save it.
Step 3: Open your new list in Microsoft Access
In your Office 365 account open your list and select the Open with Access option highlighted below.
When the list opens choose to have data linked to the sharepoint site like in the below pic.
Step 4: Copy the core data
Unfortunately you cannot copy everything all in one go because the rows must be created before an item can be attached. From here the next thing to do is to select the column headers in the access instance from the Office Live Small Business list. Select all headers except the one for the attachments column. Like in the below picture.
Next go to the Access instance which is linked to the Sharepoint List in the Office 365 site. Paste the lines into this Access instance. The paste action will slowly insert the new rows into the Office 365 SharePoint List.
Step 5: Copy Attachments
You should now have two Microsoft Access databases with the same rows in. One a copy from Office Live Small Business and one with a linked table to the Office 365 SharePoint list. The rows will all be in the same order.
The next step is to copy the attachments.
In the Office Live Small Business Access instance select the column header for the attachments. The one with the paper clip symbol as its header. Then copy the entire column.
In the Office 365 Access instance select the attachments column header and paste the entire column.
This paste action will probably take a while if you have a lot of attachments.
Conclusion
As I mentioned at the start the biggest concern I had with the Office Live Small Business to Office 365 migration was around what to do with all of my custom lists and attachments. Unfortunately the self migration guide wasn't very useful for this bit because It just said to export your data but didn't cover importing it or anything about attachments.
As you can see this only took a short time to do so generally I am quite pleased that this is complete and I can now get on with enjoying the many new features of Office 365 and let my OLSB account disappear into the past.
Sunday, January 15, 2012
I've been working on a little project recently and one of the requirements was to be able to identify someone's influence in the social media space. I happened to come across Klout which if you are not familiar is a service where people can connect all of their social networking accounts together and it will then analyze your public activity and workout how much Klout you have. Even if you have not joined Klout it still looks at public Twitter information and can rate users just based on their public Twitter activity.
This sounded like a service worth investigating because it looks at who you follow and who follows you and things like retweets and mentions. It then rates you. This sounds like a good service based way of finding out how influential a user is.
The aim of this article is as much for myself as a reminder of this simple POC just to retrieve a score.
Start off by registering your application by signing up with the Klout developer site.
http://developer.klout.com/
Next the below class can wrap up interaction with Klout. You can see in the get user ratings method I take a list of user names and then format the url and make calls to Klout. The response is loaded into a Linq to XML XDocument so I can easily extract the score. A dictionary is returned of the user ratings.
There is probably tidier ways to do this and im sure we will refactor the code etc as we go through the project but for now the little sample below may be useful to someone as I haven't seen many C# samples online.
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Net;
using System.Xml.Linq;
using System.Xml;
using System.Diagnostics;
namespace Acme.Klout
{
///
<summary>
/// Wrapper for the klout service
///
</summary>
public
class
KloutClient
{
private
string KloutKey
{
get { return
"[ADD YOUR KEY HERE]"; }
}
public
Dictionary<string, decimal> GetUserRatings(List<string> users)
{
const
string urlFormat = @"http://api.klout.com/1/klout.xml?users={1}&key={0}";
var response = new
Dictionary<string, decimal>();
var scoreDecimal = decimal.Zero;
foreach(var user in users)
{
var url = string.Format(urlFormat, KloutKey, user);
try
{
scoreDecimal = decimal.Zero;
var kloutResponse = GetKloutResponse(url);
var document = XDocument.Load(kloutResponse);
var score = document.Descendants("kscore").First().Value;
scoreDecimal = decimal.Parse(score);
}
catch (Exception ex)
{
Trace.WriteLine("Error getting klout rating: " + ex.ToString());
}
response.Add(user, scoreDecimal);
}
return response;
}
private
Stream GetKloutResponse(string url)
{
var request = WebRequest.Create(url);
var response = request.GetResponse();
return response.GetResponseStream();
}
}
}
Saturday, December 17, 2011
Ive been doing a little stuff with the Twitterize library and the twitter API.
I started unexpectedly getting a 401 unauthorized error when things had been working fine previously.
Eventually I tracked this down to be an issue with clock sync. I am using VM Ware fusion on Mac and when if suspended then resumed my VM the clock wasnt always in sync. When this was fixed it all works fine.
Note to self the way to change the VM is
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021718
Saturday, December 10, 2011
For a while I’ve been working on an idea and whitepaper around analyzing source code to measure the size and complexity. There are some tools around which can do this but my differentiator is that I was interested in BizTalk projects and wanted to look at how these could be measured. Note however this can be easily used for non-BizTalk projects too.
The aim is to be able to understand the size and complexity of a solution and to allow some real metrics to contribute to your understanding of the Total Cost of Ownership of your solution. Using the tools I have developed alongside this I have been working with some customers to integrate the analysis into the build and continuous integration processes so that we can produce a report of the code and also track changes over time to monitor how the code based vary in terms of size and complexity.
The white paper goes into the detail of the theory about how we measure code and is available on the following link:
https://skydrive.live.com/redir.aspx?cid=983a58358c675769&resid=983A58358C675769!1820&parid=983A58358C675769!1819
The source code and application/tools which can do this for you are available on the following codeplex project site
http://biztalksolutionanaly.codeplex.com/
At this stage the project is being used to monitor approximately 20 code bases at two companies who have significant investments in BizTalk and I’m also looking for feedback from the community over other things we should measure.
It would be great to hear how you get on with this or if you have any views on it.
Thursday, December 01, 2011
A while back I did the user group sessions and cloud case video about Behaviour Driven BizTalk Development and talked about how you could use SpecFlow with Visual Studio 2010, BizUnit 4 and BizTalk 2010 to create acceptance tests for your solution.
The video is on the below link:
Part 1: http://cloudcasts.cloudapp.net/ViewWebcast.aspx?webcastid=2520897495056204879
Part 2: http://cloudcasts.cloudapp.net/ViewWebcast.aspx?webcastid=2520897493960073864
Of the many advantages of this approach one of the best ones was the benefits of using acceptance tests as documentation of the solution. Well written acceptance tests can become excellent, accurate and tested documentation which sits with the code.
So while all is good in BizTalk 2010 world, I've recently been working with one of my BizTalk 2006 customers and wanted to explore how we could do something to bring these advantages to BizTalk 2006. The key limitation for BizTalk 2006 is that Specflow did not support Visual Studio 2005. A few weeks ago I talked in a separate blog article about how we could use xml comments in the BizUnit 2 xml files to be able to annotate our existing tests so that we have a structure which gives a good idea of the intention of a test to sit alongside the implementation.
http://geekswithblogs.net/michaelstephenson/archive/2011/11/04/147577.aspx
So while we are now in a better place I wanted to convert the comments from these xml files into some kind of documentation which I can easily distribute to none BizTalk developers. To solve this problem I have developed an MsBuild task which you can point to directories which contain your BizUnit tests and then it will recursively look for files in those directory which contain BizUnit tests.
When a test is identified we simply use some regular expressions to pull out the feature name, scenario name and gherkin statements I talked about previously and then produce html reports documenting each feature.
Here is an example of a documented test:
Here is an example of the usage of the msbuild task:
<!-- Gherkin Documentation of BizUnit -->
<AppFx.Build.Tasks.BizUnit.DocumentAcceptanceTests
FolderPaths="@(BizUnitFolderToDocument)"
OutPutFolder="$(PackageDirectory)\Documentation\Features"/>
You will need to do the normal importing of an MsBuild task and also setup the properties and item collection for the folders to inspect for tests.
Here is an example of the produced html report:
A couple of points to note:
- If you have some BizUnit tests which do not have the comments in then they are marked in a file canned .html which contains the undocumented tests
- We plugged the task into our MsBuild process so that everytime we produce a build of code the documentation is also included with it.
- Each html file will contain all of the documentation for scenarios which were marked with the same feature name.
Hopefully this will help lower the cost of ownership of your existing solutions and help when people join and leave your development team to help make your BizUnit tests easier to understand.
Download the bits
The code and bits for this article are available in my skydrive samples folder on the following link:
https://skydrive.live.com/?cid=983A58358C675769&id=983A58358C675769%211812
Friday, November 04, 2011
One of our .net application teams has had a problem for quite a while that related to impersonation and kerberos multi-hop delegation which had proven quite difficult to resolve. We eventually resolved this and I thought it would be worth popping a little bit of information about it out there incase anyone else has similar problems.
We had two web services with 2 methods which participate in a Kerberos multi-hop delegation scenario using WSE 2. One of the methods works fine all of the time and the other method intermittently was having problems.
Symptoms
The symptoms we experienced were:
- For method 1 everything worked fine all of the time
- Everything appeared to be working fine for method 2 but after a period of time it would start getting errors which indicated the Kerberos token had expired
- Method 2 also looked occasionally like the wrong user identity had been flown over the wire to the back end service
- The calls to method 2 which stopped working after a period of time seemed to coincide with the expiry of the Kerberos ticket (around a week to 10 days can't remember the exact time frame but it coincided with the cache duration of a Kerberos ticket)
- After an IIS reset things worked fine straight away for a period of time and then would stop working after the time period above
- The error message we were getting was: The Kerberos credential handle could not be acquired. The AcquireCredentialsHandle call returned the following error code: A specified logon session does not exist. It may already have been terminated.
Troubleshooting
This was very painful to troubleshoot, the .net team were struggling to recreate this problem in any of their test environments. Through the addition of some instrumentation they were able to derive what was going on.
What was happening
When the development team originally coded web service A for method 2 they had implemented a singleton or caching pattern of the proxy object for web service 2. This meant they did not need to construct a new instance each time.
This meant that when the method 2 was called then the WSE policy was applied and the Kerberos token used etc but this object must internally be handling things like WS-Conversation and some kind of caching so that the Kerberos token can be reused over the lifetime of that proxy instance. In essence the first user to create the object's identity was used for every subsequent usage of this object when applying a Kerberos token to the web service calls made with this proxy.
Also this would explain why after a period of time the calls fail because the Kerberos ticket obtained for that user would have expired.
Solution
The solution was to change the code in web service A so that the proxy to web service B was not cached or a singleton and so that a new instance of this proxy was created each time.
Would this apply to WCF
At this stage I have not tested if WCF would get the same problem but I do know you run into issues in various configurations if you haven't disposed of a proxy class correctly so generally it has become a very common practice for developers to safely dispose of their WCF proxies so this tends to be less of an issue. In WSE 2 or 3 developers are not used to having to use a dispose pattern around web service proxies
Lessons to learn
Some take away's for this include:
- It is a lot more likely to be your own code than framework code causing obscure problems like this
- Impersonation can be a dangerous thing so ensure you test and code review any usage of it
Credit goes to Matt Buckley who found the needle in the stack of needles in the custom .net application and found the cached proxy instance through extensive analysis of the custom code.
I've been back working with BizTalk 2006 R2 for a customer recently and I've become such a fan of the BDD style acceptance tests I've done in the past with BizTalk 2010 that its quite frustrating working back in Visual Studio 2005 and not being able to use Specflow alongside BizUnit 4 like I described in the recent videos on these subjects
In BizTalk 2006 development your back to the older style xml bizunit tests and we were looking at some old tests written a long time ago and frankly it's hard to remember what they were supposed to do. This is where the separation between intention and implementation of the test is so important. Since there isn't a great deal we can do about this in terms of upgrading the technology we took the very simple approach of annotating the BizUnit tests using the Gherkin style so that before each test step or group of test steps there was something to explain what was intended to happen. The other benefit is it really encourages the TDD approach.
In the first example below you can see when writing a new test we have simply used the Gherkin syntax Given/When/Then to record the requirements and definition of what needs to happen in this test.
<TestCase testName="VAU.CardChanges">
<TestSetup>
<!-- GIVEN: The test event queue is clean-->
<!-- GIVEN: The test folder is empty - Acquirer folders -->
<!-- GIVEN: The test folder is empty - CardSystem folders -->
<!-- GIVEN: The test folder is empty - CRM folders -->
<!-- GIVEN: The acquirer file generation numbers are zero -->
</TestSetup>
<TestExecution>
<!-- WHEN: TWS calls BizTalk to trigger the validation of payments -->
<!-- THEN: The response will indicate that validation has started -->
<!-- THEN: BizTalk will start the batch processing orchestration -->
<!-- THEN: BizTalk will send the batch to the acquirer -->
<!-- THEN: The acquirer will find the batch in their input folder -->
<!-- THEN: The acquirer will put the response in their output folder -->
<!-- THEN: BizTalk will recieve an acknowledgement from the acquirer -->
<!-- THEN: BizTalk will recieve a response file from the acquirer -->
<!-- THEN: BizTalk will process any card updates -->
<!-- THEN: The card will be updated in CARDSYSTEM-->
<!-- THEN: The validation batch will be updated in CARDSYSTEM-->
<!-- THEN: The card will be updated in CRM-->
<!-- THEN: Biztalk will have completed processing the validation batch response -->
<!-- THEN: The card update received by CRM will be valid -->
<!-- THEN: The card update received by CARDSYSTEM will be valid -->
</TestExecution>
<TestCleanup>
<!—Clean Test Folders -->
</TestCleanup>
</TestCase>
The fully completed and implemented test looks like the below example. Now hopefully you can clearly see how important this simple documentation around the tests are for people who may look at this in the future.
<TestCase testName="VAU.CardChanges">
<TestSetup>
<!-- GIVEN: The test event queue is clean-->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.CleanEvents">
</TestStep>
<!-- GIVEN: The test folder is empty - Acquirer folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerResponse</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<!-- GIVEN: The test folder is empty - CardSystem folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CardSystemWS</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<!-- GIVEN: The test folder is empty - CRM folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CRMWS</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<!-- GIVEN: The acquirer file generation numbers are zero -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.DBExecuteNonQueryStep">
<DelayBeforeExecution>1</DelayBeforeExecution>
<ConnectionString>Persist Security Info=False;Integrated Security=SSPI;Server=localhost;Database=PaymentCardServices</ConnectionString>
<SQLQuery>
<RawSQLQuery>UPDATE FileNumber SET FileNumber = 1 WHERE ID = 'NextFGN'</RawSQLQuery>
</SQLQuery>
</TestStep>
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.DBExecuteNonQueryStep">
<DelayBeforeExecution>1</DelayBeforeExecution>
<ConnectionString>Persist Security Info=False;Integrated Security=SSPI;Server=localhost;Database=PaymentCardServices</ConnectionString>
<SQLQuery>
<RawSQLQuery>UPDATE FileNumber SET FileNumber = 1 WHERE ID = 'ActiveFGN'</RawSQLQuery>
</SQLQuery>
</TestStep>
</TestSetup>
<TestExecution>
<!-- WHEN: TWS calls BizTalk to trigger the validation of payments -->
<TestStep assemblyPath="Acme.BizTalk.Fx.Testing.dll" typeName="Acme.BizTalk.Fx.Testing.BizUnit.Wse2WebServicesClientProtocolStep">
<WebServiceUrl>http://localhost/Acme.CardProcessing.Secure/CardService.ashx</WebServiceUrl>
<ProxyTypeName>Acme.CardProcessing.Utilities.Testing.TestProxies CardService, Acme.CardProcessing.Utilities</ProxyTypeName>
<ProxyMethodName>SubmitBatchValidation</ProxyMethodName>
<ExpectingError>false</ExpectingError>
<InputMessageTypeName>Acme.CardProcessing.Utilities.Testing.TestProxies.CardValidationBatch, Acme.CardProcessing.Utilities</InputMessageTypeName>
<MessagePayloadPath>..\..\..\Acme.CardProcessing.AcceptanceTests\Features\BatchValidation\TestData\CardChanges.Input.xml</MessagePayloadPath>
<!-- THEN: The response will indicate that validation has started -->
<ValidationStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.XmlValidationStep">
<XPathList>
<XPathValidation query="/ValidationBatchesReturn/Return">0</XPathValidation>
</XPathList>
</ValidationStep>
</TestStep>
<!-- THEN: BizTalk will start the batch processing orchestration -->
<TestStep assemblyPath="Acme.CardProcessing.AcceptanceTests.dll" typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Batch validation started</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: BizTalk will send the batch to the acquirer -->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Batch sent to acquirer</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: The acquirer will find the batch in their input folder -->
<TestStep assemblyPath="Acme.BizTalk.Fx.Testing.dll" typeName="Acme.BizTalk.Fx.Testing.BizUnit.FileExistsStep">
<Timeout>60</Timeout>
<DirectoryPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</DirectoryPath>
<SearchPattern>*.*</SearchPattern>
<ExpectedNoOfFiles>1</ExpectedNoOfFiles>
</TestStep>
<!-- THEN: The acquirer will put the response in their output folder -->
<TestStep assemblyPath="Acme.CardProcessing.AcceptanceTests.dll" typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.ProcessAcquirerFileStep">
<DirectoryPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</DirectoryPath>
<SearchPattern>INZET.DANG.MDNDMDM</SearchPattern>
<ResponsesXmlFileName>..\..\..\Acme.CardProcessing.AcceptanceTests\GenericTestData\HSBCDefaultBatchResponse.xml</ResponsesXmlFileName>
<SchemaFileName>..\..\..\Acme.CardProcessing.AcceptanceTests\GenericTestData\BatchResponses.xsd</SchemaFileName>
<CreationPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerResponse</CreationPath>
<InvalidResponse></InvalidResponse>
<MultipleResponse>false</MultipleResponse>
</TestStep>
<!-- THEN: BizTalk will recieve an acknowledgement from the acquirer -->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Acknowledgement received from acquirer</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: BizTalk will recieve a response file from the acquirer -->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Acquirer response received</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: BizTalk will process any card updates -->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Processing card updates</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: The card will be updated in CARDSYSTEM-->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>CardServicesStub.CardServices.UpdatePaymentCards Called</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: The validation batch will be updated in CARDSYSTEM-->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>CardServicesStub.CardServices.UpdateValidationBatch Called</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: The card will be updated in CRM-->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>CRMApplicationStub.FinanceRefService.UpdateCreditCardDetails Called</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: Biztalk will have completed processing the validation batch response -->
<TestStep
assemblyPath="Acme.CardProcessing.AcceptanceTests.dll"
typeName="Acme.CardProcessing.AcceptanceTests.BizUnitSteps.TestEvents.EventExists">
<Action>Batch validation completed</Action>
<RemoveEventFromQueue>true</RemoveEventFromQueue>
<SecondsToWait>30</SecondsToWait>
</TestStep>
<!-- THEN: The card update received by CRM will be valid -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileValidateStep">
<Timeout>6000</Timeout>
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CRMWS</Directory>
<SearchPattern>*.*</SearchPattern>
<DeleteFile>false</DeleteFile>
<ValidationStep assemblyPath="Acme.CardProcessing.Utilities.dll" typeName="Acme.CardProcessing.Utilities.Testing.BizUnit.XmlComparisonValidationStep">
<CompareTo>CardChanges.CRM.UpdateCards.xml</CompareTo>
</ValidationStep>
</TestStep>
<!-- THEN: The card update received by CARDSYSTEM will be valid -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileMultiValidateStep">
<Timeout>6000</Timeout>
<DirectoryPath>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CardSystemWS</DirectoryPath>
<SearchPattern>*.*</SearchPattern>
<ValidationStep assemblyPath="Acme.CardProcessing.Utilities.dll" typeName="Acme.CardProcessing.Utilities.Testing.BizUnit.XmlComparisonValidationStep">
<CompareTo>CardChanges.CardSystem.UpdateCards.xml</CompareTo>
</ValidationStep>
<ValidationStep assemblyPath="Acme.CardProcessing.Utilities.dll" typeName="Acme.CardProcessing.Utilities.Testing.BizUnit.XmlComparisonValidationStep">
<CompareTo>CardChanges.UpdateBatch.xml</CompareTo>
</ValidationStep>
</TestStep>
</TestExecution>
<!-- Clear out all test folders -->
<TestCleanup>
<!-- Acquirer folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerRequest</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\AcquirerResponse</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<!—Card App folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CardSystemWS</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
<!-- CRM folders -->
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
<Directory>..\..\..\Acme\CardProcessing\Stubs\DropFolders\CRMWS</Directory>
<SearchPattern>*.*</SearchPattern>
</TestStep>
</TestCleanup>
</TestCase>
This is just a reminder for my self of how and why we do this.
We have 2 projects within a TFS project collection for our integration component developments. We have:
1. A project for .net based integration projects
2. A project for BizTalk based integration projects
The main reason we do this is so we dont have loads of TFS projects as we have a significant number of components but also we want some different rules around check in and source control locks etc etc.....
At this stage our build servers are still running cruise control rather than Team Build so one of the problems i was finding was when cruise control labelled a build in TFS i couldnt tell easily in the label search which component or folder the build label related to. We modified the build scripts to do a custom label with TF.exe so that we could put a specific label from each build. We used the following command:
<Exec Command='"C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\tf" label "$(CCNetProject).$(CCNetLabel)" $(MsBuildProjectDirectory) /recursive'/"/>
Using this command means that the build label is applied at the right place for the workspace but also the label includes the cruise control project name making it easy to workout what the label was actually for rather than just being a bunch of version numbers.
There is probably a better way to do this but its working fine for us.
Wednesday, October 19, 2011
We are currently evaluating BizTalk 360 at one of my clients at the moment and I plan to write a more detailed post in the future about our experiences but in the meantime I wanted to make a comment about something this week which was particularly useful.
We have a very large project with many teams and vendors and in our first integration test environment all of the delivery teams do daily deployments to this test environment. From here a successfully tested release could be promoted to other environments as required. Its quite painful at times due to the pace of change in this first test environment and im sure most readers would be familiar with the typical challenges around integration and biztalk being the bit that identifies the problems and often not be the cause of the problem. Yes you know what I am talking about.
A significant amount of my teams time is taken up with troubleshooting issues and working out retrospectively where a problem came from. In our evaluations of BizTalk 360 one of the things I have done is to setup the BizTalk 360 health check emails to be published 4 times per day.
This week we had a situation where the test teams had signed off a release but had not realised for some reason there was an issue where some messages were unable to be loaded into an application. This whole release had been signed off to go into the UAT environment and then an hour later the BizTalk 360 email came out and we could see these suspended messages. We were able to identify and rectify this issue in the application which was not accepting messages and plug the gap in the test scripts before the release went to UAT.
Finding this issue in the system test environment and resolving it would have saved a significant impact with the problems this issue could have caused in the UAT environment in terms of lost test time etc etc. In my opinion this one situation probably paid for the cost of the BizTalk 360 license on its own for our large scale project so hopefully our evaluation will be finished soon and we will be able to roll this out across other environments.
Thanks Saravana keep up the good work
Friday, August 05, 2011
The biztalk videos for BDD and acceptance testing are now on cloudcast
http://www.cloudcasts.net/Default.aspx?category=BizTalk
Tuesday, July 19, 2011
I always seem to spend ages looking for this when i need it so this is the link to terminator
http://www.microsoft.com/download/en/details.aspx?id=2846
Monday, July 18, 2011
Ive recently done some user group sessions around BizTalk and Behaviour Driven Development and Acceptance Testing.
Ive uploaded the videos and samples for these sessions to the following codeplex site.
http://biztalkbddsample.codeplex.com/
They will also be on CloudCasts soon
Enjoy
Saturday, June 11, 2011
Ive just released an update to the codeplex project so that there are test steps which are also compatible with BizUnit v4 which has recently been released
http://btsloggingeventsinbi.codeplex.com/
Friday, June 03, 2011
I was chatting the other day with someone about adapters for connecting to LOB applications and an interesting point came up which I thought id share my thoughts on.
The scenarios is that if you have a line of business application for arguments sake lets say its dynamics CRM which has a BizTalk adapter available but also has an existing web service API (or some other protocol based API). Which should you use for integration? In my opinion the answer to this is the usual "it depends" answer. I think what is particularly important are some of the considerations which you should make when deciding your approach.
I thought id put down a few thoughts from my past experience and see where we end up.
1. Testability
In order to deliver a high quality integration solution its important to be able to test your solution very effectively. With some line of business applications if there is a standards based interface and an open protocol then its usually fairly easy to stub out the dependent system making it easy to test your solution in isolation.
If you have an application adapter then this can often create a partial coupling to the line of business system for your testing. There are techniques that BizTalk people can use to handle this such as changing to a different adapter for some BizUnit tests in the development environment to allow you to do this testing.
With a little up front thinking you can usually plan how your development process will work in relation to this dependency but I have seen projects where they ended up with a problem because the integration team could not work because of the dependency on a line of business application which was also in development at the same time and the application developers had broken something which was a blocker for the integration team.
The key point here is that a protocol adapter probably makes it easier to stub out your dependencies.
2. Dependencies
Application adapters will often require some kind of client side components to be installed on the BizTalk server. Protocol adapters are usually unlikely to have many client side requirements.
You should be aware of these dependencies and ensure you can install and manage them and that there are no conflicts with other software you need.
I remember once there was a project where we were integrating into both Cognos and Teradata and the client side dependencies of the vendor components we were using to connect to these systems were incompatible meaning the two integration components could not co-exist on the same server.
3. Vendor support
If your using a protocol adapter then in most cases you are probably using the normal out of the box BizTalk ones which will be part of your existing BizTalk support agreement. You will also be likely to have an agreement with the vendor for your application about the API they expose. In this case unless you have a less common protocol then for most cases your normal support agreements will cover things.
In the case of an application adapter then you need to consider who is the vendor for the adapter. If it's a Microsoft adapter then your probably in the same situation as with your normal BizTalk support agreement. If the vendor is a third party or the same as the vendor for the application then you would be likely to need additional support costs and considerations. In the case when the adapter vendor is a 3rd party you often will have a partner who is an expert in this kind of integration but also you could find it difficult to workout if a problem is BizTalk itself, the adaptor or the application.
The support position should be an important consideration.
4. Vendor Technology Road maps
Its important to be aware of the technology road maps for the application, BizTalk and any adapter vendor.
In the case of a normal protocol adapter there is probably only two parties involved and the communication would usually be standards based so it should be easy to workout any compatibility issues. You then just have the challenge of a contract-based interface to manage and any information about breaking behavior of contract changes.
The application adapter could be a much more complicated scenario. You need to think about the following:
- If I upgrade the LOB application will it work with my existing adapter
- If I upgrade my LOB application and need to update the adapter will this work with my version of BizTalk
- If I upgrade BizTalk will the adapter continue to work with the new version of BizTalk
I think in the application adapter scenario you need to have a much closer a alignment between the road maps in your organization for compatibility of BizTalk, the adapter and the LOB application.
5. Current Organisation Approaches
You should think about if your organization already integrates with the application in a certain way from outside of BizTalk. If it does and there are similarities then you can have a certain degree of confidence in what to expect. You will also have in place some processes to manage this.
If you find your organization has an existing way of doing this that works well and fits with the strategy then definitely follow it. If the existing way doesn't work then definitely look to change it.
6. Cost
Adapters which aren't free from Microsoft or included with BizTalk are likely to cost something.
This is an obvious factor to consider. But also consider the costs of not buying an adapter. It's much harder to quantify something you build but the long term cost of ownership may be much higher than the original vendor purchase.
7. Assumptions
When you connect to an application you are making assumptions about the contract based on something like wsdl if it's a web service interface. You are also making assumptions about the behavior of the API usually based on the documentation which comes from the application vendor.
Typically when we integrate with an application using a protocol adapter I would normally look to develop some tests, which will flex the API without any of the stuff I will build on top of it. These tests will really prove my assumptions about how the application will work and potentially give me an indication of performance. When I am using a protocol adapter this kind of test is usually relatively easy to write in .net code even if the protocol adapter ends up being something from a 3rd party.
When I use a BizTalk application adapter it is likely to be a little bit harder to validate these assumptions. I can still do it but I will have to build a fairly simple BizTalk application which will allow me to push some messages through the adapter and prove how the interface will work. Its usually just more time consuming to develop tests in this way.
One point to note is that if the adapter is a WCF LOB adapter then it is often possible to use this adapter outside of BizTalk so you can potentially write the same .net tests mentioned above but use a WCF LOB adapter directly from .net code.
The key thing with this section is that it's a good idea to prove how the interface will work before you start using it so you can reduce the integration issues and mitigate the risk that you could have problems with the assumptions you make about the vendor API.
8. Feature set
With a BizTalk application adapter there is usually a reason why they have been built. It is important to be clear on the feature set offered by the adapter and what it offers above just using a protocol adapter.
In some circumstances you will have the situation where a BizTalk adapter was built and then a new version of the LOB application comes out and part of this new version includes an improved API which is more standards based and other good things. The adapter may still be compatible.
In other situations there will be clear features offered by the application adapter which you would either have to develop yourself or things which just improve the integration experience.
Carefully evaluate the features offered by each adapter to workout what you get extra and how valuable it is.
9. Buy Versus Build
Some organisations have a clear buy versus build strategy. Without going into too much detail this could significantly influence your decision.
10. Be Open Minded
One of the things that can be quite frustrating is when a design decision is made simply because that's the way its always been done and there is fear of change or something that the team do not understand. Just because you always use a certain way to connect to a given application doesn't mean a different way does not have its own merits and might not be better. Particularly if a new version of an adapter comes out.
On this point I would say don't always jump to your default choice. At the very least try out the other options.
Summary
As I said at the start of this article there is no one answer for every situation, but in practice I often find that the application adapter could be more work or cost to your solution. In this case I usually want to have clear justification for why we are using the application adapter over the protocol adapter so that we have a well defined design decision for our project which is also inline with our integration strategy.
This was a bit of a brain dump from my experience and it would be interesting to see if others agree or disagree with my thoughts.
Tuesday, May 17, 2011
A few weeks ago I did the video about how we were testing what happens inside BizTalk by using the CAT Team Logging Framework and the ETW trace events and then testing the information coming out from our tracing to prove what was happening inside BizTalk.
Ive updated the codeplex project so that the code is a little easier for people to use and is packaged better. Also the source code is now available in codeplex and there is some additional documentation.
Note that it now works slightly differently to in the video but performs much better. The original video about the idea is on the following link
http://www.cloudcasts.net/ViewWebcast.aspx?webcastid=2521028145037920619
The codeplex project is on the following link:
http://btsloggingeventsinbi.codeplex.com/
Friday, May 13, 2011
Ive recently just setup BizTalk 360 (http://www.biztalk360.com/) to try out the monitoring capabilities. I also thought it would be interesting to see how things go since this given environment was a BizTalk 2006R2 environment and fairly old stuff now.
I had a couple of issues during the install/setup but i was able to solve all of these using the troubleshooting page on the BizTalk 360 website and a little common sense. To be honest a couple of them were me being a bit lazy and not reading the install notes properly.
The changes I had to make were as follows:
1. Changed connection string to the biztalk 360 database as described on the troubleshooting website to integrated security
2. Change the IIS6 virtual directory to .net 4 rather than .net 2
3. Add the full trust element to the web.config file for BizTalk 360
4. Add the IIS NT Authentication providers as described on the troubleshooting page
Bearing in mind this was a CTP release I was very pleased with the support on the website which helped me to get this setup and even with these problems it still took less than 20 minutes. I would expect BizTalk 2006 to be a slightly less target for the BizTalk 360 team than BizTalk 2010 but it goes to prove it can be installed pretty easily and im not enjoying the management features which I think are a big opportunity for lots of organisations to really get their production operations of biztalk working in secure and effective way
Good job guys
Saturday, April 23, 2011
Ive eventually had time to migrate the build generator to work with BizTalk 2010.
Ive released this as a beta for now and would love to hear any feedback from others who might be using it while i work on getting some more testing with other migrated projects.
If your interested in taking a look please refer to:
http://biztalkmsbuildtool.codeplex.com/releases/view/65036
Friday, April 15, 2011
The next uk connected systems user group event is open for registration:
11th May 2011 @ EMC
http://ukcsbugmay2011.eventbrite.com/
Friday, March 25, 2011
Ive just posted a paper on the technet wiki about BizTalk RFID and NServiceBus used as a publishing component
http://social.technet.microsoft.com/wiki/contents/articles/biztalk-2010-amp-nservicebus.aspx
Tuesday, March 08, 2011
Just to let everyone know the BizTalk Light and Easy Webcast Series has been updated with some new BizTalk 2010 videos
http://www.cloudcasts.net/Default.aspx?category=BizTalk+Light+and+Easy
Tuesday, March 01, 2011
As BizTalk developers we all know that one of the common challenges is how to test your BizTalk solution once it is deployed to BizTalk. Hopefully most of us are using the BizUnit framework for testing, but we still have the limitation that it's a very Black Box test.
I have put together a sample and video to show a technique where I'm using the Logging Framework from the BizTalk CAT Team at Microsoft and where by BizUnit test is able to make assertions against the instrumentation going through the framework. This means that I can test for things happening such as the fact a component was executed or which branch of an orchestration was executed by simply using my normal instrumented code.
I've put the sample and video for this on the following codeplex site:
http://btsloggingeventsinbi.codeplex.com/
The video should also be on cloud casts fairly soon too.
Saturday, January 29, 2011
I have been chatting with Maurice den Heijer recently about his codeplex project for the BizTalk Map Testing Framework (http://mtf.codeplex.com/).
Some of you may remember the article I did for BizTalk 2009 and 2006 about how to test maps but with Maurice's project he is effectively looking at how to improve productivity and quality by building some useful testing features within the framework to simplify the process of testing maps.
As part of our discussion we realized that we both had slightly different approaches to how we validate the output from the map. Put simple Maurice does some xpath validation of the data in various nodes where as my approach for most standard cases is to use serialization to allow you to validate the output using normal MSTest assertions.
I'm not really going to go into the pro's and con's of each approach because I think there is a place for both and also I'm sure others have various approaches which work too. What would be great is for the map testing framework to provide support for different ways of testing which can cover everything from simple cases to some very specialized scenarios.
So as agreed with Maurice I have done the sample which I will talk about in the rest of this article to show how we can use the serialization approach to create and compare the input and output from a map in normal development testing.
Prerequisites
One of the common patterns I usually implement when developing BizTalk solutions is to use xsd.exe to create .net classes for most of the schemas used within the solution. In the testing pattern I will take advantage of these .net classes.
The Map
In this sample the map we will use is very simple and just concatenates some data from the input message to the output message. Hopefully the below picture illustrates this well.
The Test
In the test I'm basically taking the following actions:
- Use the .net class generated from the schema to create an input message for the map
- Serialize the input object to a file
- Run the map from .net using the standard BizTalk test method which was generated for running the map
- Deserialize the output file from the map execution to a .net class representing the output schema
- Use MsTest assertions to validate things about the output message
The below picture shows this:
As you can see the code for this is pretty simple and it's all strongly typed which means changes to my schema which can affect the tests can be easily picked up as compilation errors. I can then chose to have one test which validates most of the output from the map, or to have many specific tests covering individual scenarios within the map.
Summary
Hopefully this post illustrates a powerful yet simple way of effectively testing many BizTalk mapping scenarios.
I will probably have more conversations with Maurice about these approaches and perhaps some of the above will be included in the mapping test framework.
The sample can be downloaded from here:
http://cid-983a58358c675769.office.live.com/self.aspx/Blog%20Samples/More%20Map%20Testing/MapTestSample.zip
Friday, January 28, 2011
Im currently in the process of migrating our many BizTalk projects from MKS source control to TFS. While we will be using TFS for work item tracking and source control etc we will be continuing to use Cruise Control for continuous integration although im updating this to CCNet 1.5 at the same time.
Ill post a few things as much as a reminder to myself about some of the problems we come across.
Problem
After the first build of our code the next time a build is triggered an error is encountered by the TFS source control block refreshing the source code.
System.IO.IOException: The directory is not empty.
at System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive)
at System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive)
at ThoughtWorks.CruiseControl.Core.Sourcecontrol.Vsts.deleteDirectory(String path)
at ThoughtWorks.CruiseControl.Core.Sourcecontrol.Vsts.GetSource(IIntegrationResult result)
at ThoughtWorks.CruiseControl.Core.IntegrationRunner.Build(IIntegrationResult result)
at ThoughtWorks.CruiseControl.Core.IntegrationRunner.Integrate(IntegrationRequest request)
Solution
The problem seems to be with a folder called TestLocations which is created by the build process and used along with the file adapter as a way to get messages into BizTalk. For some reason the source control block when it does a full refresh of the code does not get rid of this folder and then complains thats a problem and fails the build.
Interestingly there are other folders created by the build which are deleted fine. My assumption is that this if something to do with the file adapter polling the directory. However note that we have not had this problem with other source control blocks in the past.
To workaround this I have added a prebuild task to the ccnet.config file to delete this folder before the source control block is executed. See below for example
prebuild> exec>executable>cmd.exe</executable>buildArgs>/c "if exist "C:\<MyCode>\TestLocations" rd /s /q "C:\<MyCode>\TestLocations""</buildArgs>exec> prebuild>
<
Saturday, January 15, 2011
Ive just recently done a video about BizTalk and Visual Studio 2010 Layer Diagrams which Alan Smith has kindly published on cloudcasts
http://www.cloudcasts.net/ViewWebcast.aspx?webcastid=2521072060555401694
Tuesday, December 28, 2010
To follow up a previous article about timeouts and how they can affect your application I have extended the sample we were using to include WCF. I will execute some test scenarios and discuss the results.
The sample
We begin by consuming exactly the same web service which is sitting on a remote server. This time I have created a .net 3.5 application which will consume the web service using the basichttp binding. To show you the configuration for the consumption of this web service please refer to the below diagram.
You can see like before we also have the connectionManagement element in the configuration file.
I have added a WCF service reference (also using the asynchronous proxy methods) and have the below code sample in the application which will asynchronously make the web service calls and handle the responses on a call back method invoked by a delegate.
If you have read the previous article you will notice that the code is almost the same.
Sample 1 – WCF with Default Timeouts
In this test I set about recreating the same scenario as previous where we would run the test but this time using WCF as the messaging component. For the first test I would use the default configuration settings which WCF had setup when we added a reference to the web service.
The timeout values for this test are:
closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
The Test
We simulated 21 calls to the web service
Test Results
The client-side trace is as follows:
The server-side trace is as follows:
Some observations on the results are as follows:
- The timeouts happened quicker than in the previous tests because some calls were timing out before they attempted to connect to the server
- The first few calls that timed out did actually connect to the server and did execute successfully on the server
Test 2 – Increase Open Connection Timeout & Send Timeout
In this test I wanted to increase both the send and open timeout values to try and give everything a chance to go through.
The timeout values for this test are:
closeTimeout="00:01:00" openTimeout="00:10:00" receiveTimeout="00:10:00" sendTimeout="00:10:00"
The Test
We simulated 21 calls to the web service
Test Results
The client side trace for this test was
The server-side trace for this test was:
Some observations on this test are:
- This test proved if the timeouts are high enough everything will just go through
Test 3 – Increase just the Send Timeout
In this test we wanted to increase just the send timeout.
The timeout values for this test are:
closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:10:00"
The Test
We simulated 21 calls to the web service
Test Results
The below is the client side trace
The below is the server side trace
Some observations on this test are:
Test 4 – Increase Just the Open Connection Timeout
In this test I wanted to validate the change to the open connection setting by increasing just this on its own.
The timeout values for this test are:
closeTimeout="00:01:00" openTimeout="00:10:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
The Test
We simulated 21 calls to the web service
Test Results
The client side trace was
The server side trace was
Some observations on this test are:
- In this test you can see that the open connection which relates to opening the channel timeout increase was not the thing which stopped the calls timing out
- It's the send of data which is timing out
- On the server you can see that the successful few calls were fine but there were also a few calls which hit the server but timed out on the client
- You can see that not all calls hit the server which was one of the problems with the WSE and ASMX options
Test 5 – Smaller Increase in Send Timeout
In this test I wanted to make a smaller increase to the send timeout than previous just to prove that it was the key setting which was controlling what was timing out.
The timeout values for this test are:
openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:02:30"
The Test
We simulated 21 calls to the web service
Test Results
The client side trace was
The server side trace was
Some observations on this test are:
- You can see that most of the calls got through fine
- On the client you can see that call 20 timed out but still hit the server and executed fine.
Summary
At this point between the two articles we have quite a lot of scenarios showing the different way the timeout setting have played into our original performance issue, and now we can see how WCF could offer an improved way to handle the problem. To summarise the differences in the timeout properties for the three technology stacks:
ASMX
The timeout value only applies to the execution time of your request on the server. The timeout does not consider how long your code might be waiting client side to get a connection.
WSE
The timeout value includes both the time to obtain a connection and also the time to execute the request. A timeout will not be thrown as an error until an attempt to connect to the server is made. This means a 40 second timeout setting may not throw the error until 60 seconds when the connection to the server is made. If the connection to the server is made you should be aware that your message will be processed and you should design for this.
WCF
The WCF send timeout is the setting most equivalent to the settings we were looking at previously. Like WSE this setting the counter includes the time to get a connection as well as the time to execute on a server. Unlike WSE and ASMX an error will be thrown as soon as the send timeout from making your call from user code has elapsed regardless of whether we are waiting for a connection or have an open connection to the server. This may to a user appear to have better latency in getting an error response compared to WSE or ASMX.