BizTalk

ESB Toolkit Clients

charlie.mott — Mon, 25 Jul 2011 22:50:53 GMT

Article Source: http://geekswithblogs.net/charliemott

What is the best approach for developing a WCF client application that sends messages to the WCF on-ramps exposed by the BizTalk ESB Toolkit?

I had considered various approaches:

Generate an xml message from a string template using string replacements. Then submit the message to the ESB endpoint using code similar to the Itinerary Test Client provided with the ESB Toolkit. Then for the response use XPath to get the required data.
- This is obviously a lot of coding effort, inflexible and difficult to maintain.
Modifying the generic wsdl for the ESB WCF service as described here.
- However, this is too much of an administrative nightmare. You need to create a wsdl for each possible service method. In addition it does not support the catching of expected SOAP faults without additional effort.
Our team then considered an approach similar to that described here. This uses a generic ServiceContract interface that can take any object and serialise it to a System.ServiceModel.Channels.Message.
- However, the problem is that developers will still need to manually generate the DTO classes using a tool such as svcutil (getting the schemas from the location specified in the wsdl). As such, there is still a lot of effort required.
A colleague then asked "why don't you just add a service reference to the end service and then re-point the client endpoint config to the ESB on-ramp". I had previously dismissed this idea because the end service exposes many methods\actions while the ESB on-ramps expose just one with a different name. However, he suggested you should be able to alter the action.

Using a WCF Message Inspector

Option 4 is the best option I think. You can modify the Action header using a custom behaviour with a custom Message Inspector following the approach described here.

The modified message inspector code for the request-response on-ramp would be as follows:

        public object BeforeSendRequest(ref System.ServiceModel.Channels.Message request, IClientChannel channel)
        {
            request.Headers.Action = "http://microsoft.practices.esb/ProcessRequestResponse";
            return null;
        }

The client service code would be something like this:

        public int FindProducts (ProductSearchCommand searchCommand)
        {
            var client = new ProductServiceClient(this.EsbOnRampRequestResponseEndpointConfig);
            client.Endpoint.Behaviors.Add(new OnRampRequestResponseBehavior()); // this could be done via endpoint config (see here)
            ProductsDocument products = client.Find(searchCommand);
            client.Close();
            return products;
        }

The client direct call endpoint config would be modified to something like this to point to the ESB:

      <endpoint address="https://MyMachine/ESB.ItineraryServices.Generic.Response.WCF/ProcessItinerary.svc"
                binding="wsHttpBinding"
                bindingConfiguration="WSHttpBinding_ITwoWayAsync"
                contract="ProductService.ProductService"
                name="EsbOnRamp_RequestResponse">
        <identity>
          <userPrincipalName value="MyMachine\BizTalkUser" />
        </identity>
      </endpoint>

Notes

In BizTalk, for the WCF receive location for the default ESB on-ramp has been modified so that the WCF adapter Message “UseBodyPath” settings use "Body" for the "Inbound BizTalk message body" and "Outbound WCF message body".
The default on-ramp uses wsHttpbinding. This binding is based on SOAP 1.2. As such, soap faults will be returned using a soap 1.2 namespaces. If the end service uses basicHttpBinding, then the client code will not be able to catch the soap 1.1 fault. As such, mixing soap versions for the on-ramp and services called by an itinerary would require some additional research.

Sending header details in messages to Dynamics AX 2012 WCF services from BizTalk

charlie.mott — Fri, 06 May 2011 16:04:50 GMT

Article Source: http://geekswithblogs.net/charliemott

How do you send "Call Context" information in the header message to Dynamics AX 2012 WCF services from BizTalk?

One difference between AX 2009 and AX 2012 services, is that you no longer always need to provide destination endpoint context information. This is described here:

In previous releases, each AIF endpoint was associated with a specific company. Microsoft Dynamics AX 2012 does not require that you associate integration ports with a specific company. You can use the integration port functionality to restrict service calls to a specific company. For an inbound message, the services framework obtains the company from the message header. If the message header does not contain a company context, the services framework uses the default company for the submitting user.

Adding WCF.OutboundCustomHeaders

In my requirement, I did need to send the Company code.

There is a very good article here about making BizTalk send header information to AX 2009 using WCF Services.

However, the implementation is slightly different with AX 2012. The schema namespace you need is http://schemas.microsoft.com/dynamics/2010/01/datacontracts

So the code will look like this. Notice I have only provided the header context information needed:

wcfHeader = System.String.Format(@"<headers><CallContext xmlns=""http://schemas.microsoft.com/dynamics/2010/01/datacontracts""><Company>{0}</Company></CallContext></headers>", companyCode);

msgRequest(WCF.OutboundCustomHeaders) = wcfHeader;

Other Tips

You may need to increase the size of the WCF-NetTcp adapter "Maximum receive message size". Otherwise, if the responses are large, you may get a System.ServiceModel.CommunicationException: The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error.

Catching Faults from Dynamics AX 2012 WCF services

charlie.mott — Mon, 04 Apr 2011 23:54:45 GMT

Article Source: http://geekswithblogs.net/charliemott

I hit problems in an orchestration consuming WCF services exposed by Microsoft Dynamics AX 2012 (AX6). When attempting to catch a fault response, I was getting the error "received unexpected message type 'http://www.w3.org/2003/05/soap-envelope#fault'"

In order to fix this, I needed to change the schema for the fault message in the port type that had been created using "Add Generated Items." I changed it to the SOAP 1.2 fault type.

Thanks to this blog for pointing me in this direction:
http://masteringbiztalkserver.wordpress.com/2010/11/21/catching-soap-faults-from-wcf-service-in-biztalk-orchestration/

Test Categories with BizUnit tests

charlie.mott — Fri, 01 Apr 2011 22:06:32 GMT

Article Source: http://geekswithblogs.net/charliemott

Often it is handy to organise BizUnit tests into test lists. This way longer running tests and edge case tests can be removed from check-in builds keep them running a bit more quickly. However, updating a .vsmdi test lists for each individual test method is time consuming.

A better solution is to use the new Category attribute that comes with MSTest in .NET4 (nUnit has supported a categories feature for a while). In the TFS build processes, the mstest command line can make use of the /category flag. The check-in build will run tests categorised as Happy Flow. A Full build (nightly build) will run the lot. (Note, it is possible to do specify multiple categories).

In my solution, I have created constants for the followoing categories:

Acme.TestCategory.TestType.WakeUpBizTalk
(e.g tests where we just fire messages through biztalk without checking the results.)

Acme.TestCategory.SourceSystem.DyanmicsAx
Acme.TestCategory.SourceSystem.DynamicsCrm
Acme.TestCategory.SourceSystem.Maximo
etc

Acme.TestCategory.TargetSystem.DyanmicsAx
Acme.TestCategory.TargetSystem.Icon
etc

Acme.TestCategory.UseCase.HappyFlow
Acme.TestCategory.UseCase.AlternativeFlow (e.g. fault scenarios)

Acme.TestCategory.DataType.CustomerServices
Acme.TestCategory.DataType.Journal
etc

So a sample test method with these attributes would be as follows. Notice the multiple target systems.

/// <summary>
/// Test valid Tax Codes message from Dynamics AX
/// </summary>
[TestMethod]
[TestTimer(5, 0)]
[TestCategory(SourceSystem.DynamicsAx)]
[TestCategory(TargetSystem.Maximo)]
[TestCategory(TargetSystem.Oms)]
[TestCategory(DataType.TaxCodes)]
[TestCategory(UseCase.HappyFlow)]
public void DynamicsAx_TaxCodes_Valid_Success()
{
...

Note: For details of the TestTimer attribute, see here. This is also a handy feature for BizUnit tests to validate performance.

Creating typed WSDL’s for generic WCF services of the ESB Toolkit

charlie.mott — Mon, 20 Dec 2010 22:48:01 GMT

source: http://geekswithblogs.net/charliemott

Update (10/06/2011): I no longer recommend the approach below. It is too much of an administrative nightmare to create a wsdl for each possible service method call. See new advise here: http://geekswithblogs.net/charliemott/archive/2011/07/25/esb-toolkit-clients.aspx.

Question

How do you make it easy for client systems to consume the generic WCF services exposed by the ESB Toolkit using messages that conform to agreed schemas\contracts?

Usually the developer of a system consuming a web service adds a service reference using a WSDL. However, the WSDL’s for the generic services exposed by the ESB Toolkit use messages with type="xsd:anyType". Using these do not make it easy for the client system to use the input\output messages that conform to agreed schemas\contracts.

Recommendation

Take a copy of the generic WSDL’s and modify it to use the proper contracts.

This is very easy. It will work with the generic on ramps so long as the <part>?</part> wrapping is removed from the WCF adapter configuration in the BizTalk receive locations.

Attempting to create a WSDL where the input and output messages are sent/returned with a <part> wrapper is a nightmare. I have not managed it.

Consequences

I can only see the following consequences of removing the <part> wrapper:

ESB Management Portal - unless you intend to modify the MessageResubmitter.cs code and bindings, do not implement the above change to the "OnRamp.Itinerary.WCF" receive location. This is used by the portal to resubmit messages using WCF.
ESB Test Client – I needed to modify the out-of-the-box ESB Test Client source code to make it send non-wrapped messages.
Flat file formatted messages – the endpoint will no longer support flat file message formats. However, even if you needed to support this integration pattern through WCF, you would most-likely want to create a separate receive location anyway with its’ own independently configured XML disassembler pipeline component.

Instructions

These steps show how to implement a request-response implementation of this.

WCF Receive Locations

In BizTalk, for the WCF receive location for the ESB on-ramp, set the adapter Message settings\bindings to “UseBodyPath”:
- Inbound BizTalk message body = Body
- Outbound WCF message body = Body

Create a WSDL’s for each supported integration use-case

Save a copy of the WSDL for the WCF generic receive location above that you intend the client system to use. Give it a name that mirrors the interface agreement (e.g. Esb_SuppliersSearchCommand_wsHttpBinding.wsdl).
Add any xsd schemas files imported below to this same folder.
Edit the WSDL to import schemas

For example, this:

<xsd:schema targetNamespace=http://microsoft.practices.esb/Imports />

… would become something like:

<xsd:schema targetNamespace="http://microsoft.practices.esb/Imports">
    <xsd:import schemaLocation="SupplierSearchCommand_V1.xsd"
                          namespace="http://schemas.acme.co.uk/suppliersearchcommand/1.0"/>
    <xsd:import schemaLocation="SuppliersDocument_V1.xsd"
                            namespace="http://schemas.acme.co.uk/suppliersdocument/1.0"/>
</xsd:schema>

Modify the Input and Output message

For example, this:

<wsdl:message name="ProcessRequestResponse_SubmitRequestResponse_InputMessage">
<wsdl:part name="part" type="xsd:anyType"/>
</wsdl:message>
<wsdl:message name="ProcessRequestResponse_SubmitRequestResponse_OutputMessage">
<wsdl:part name="part" type="xsd:anyType"/>
</wsdl:message>

… would become something like:

<wsdl:message name="ProcessRequestResponse_SubmitRequestResponse_InputMessage">
<wsdl:part name="part"
                      element="ssc:SupplierSearchCommand"
                      xmlns:ssc="http://schemas.acme.co.uk/suppliersearchcommand/1.0" />
</wsdl:message>
<wsdl:message name="ProcessRequestResponse_SubmitRequestResponse_OutputMessage">
<wsdl:part name="part"
                     element="sd:SuppliersDocument"
                     xmlns:sd="http://schemas.acme.co.uk/suppliersdocument/1.0"/>
</wsdl:message>

This WSDL can now be added as a service reference in client solutions.

Naming Standards for BizUnit Integration Tests

charlie.mott — Mon, 05 Jul 2010 18:45:34 GMT

source: http://geekswithblogs.net/charliemott

Roy Osherove on his blog and in his book gives guidance on the naming of unit test methods. For use with BizUnit end-to-end integration tests, I have extended these recommendations below. Implementing these conventions has the following benefits:

Makes it easily to understand the purpose of the test.
Make it easier to find specific tests.
Gives a visual feel for integration use case test coverage.

Hub-and-Spoke Solutions

For hub-and-spoke solutions, an integration use case can typically be identified by the source system and the message type. As such, the following pattern is recommended.

Format: SourceSystem_MessageType_MessageScenarioProperties_ExpectedBehaviour()

Sample: Maximo_Invoice_Valid_Success()
Sample: Maximo_Invoice_InvalidStructure_ValidationExceptionHandled()

Integration use cases that implement a convoy pattern may need a slightly different structure. There may be different source systems and\or different message types. If the source system is different, replace with the orchestration name. If the message types are different, exclude message type details.

Format: ConvoyOrchestrationName_Scenario_ExpectedBehaviour()

Sample: JournalConvoy_Valid_Success()
Sample: JournalConvoy_SingleValidHeaderMessage_TimeoutExceptionHandled()

ESB Toolkit Solutions

In ESB Toolkit solutions, an integration use case can typically be identified by the rules that are used to select which itinerary to use. A rule might evaluate: an xpath value (message type, status value, message type version); receive location (WCF, File); etc.

This variety makes it more difficult to enforce consistency in test names. As such, the following is recommended. The test method names are prefixed with the message type for the same reasons as above. Rules are delimited by "And". The rule list excludes the prefixed source system and message type. Try to be consistent in the ordering of the rules. e.g. (1) Version (2) Status.

Format: MessageType_Action_OnRampLocationType_MessageStatus_ExpectedBehaviour

Sample: Customer_Create_WsHttp_Valid_Success()

Remember, you can write more fine-grained BizUnit tests for the BRE itinerary selection rules without re-testing the whole itinerary. See the Business Rule Engine BizUnit Test Steps codeplex project for guidance here. Also see Mike Stepheson's blog article on XPath query testing.

Code Analysis

I realise that the above recommended conventions breech the Microsoft Code Analysis Naming rule "CA1707: Identifiers should not contain underscores". However, I feel it is worth disabling this rule for test projects to gain the benefits described above.

BizTalk Cross Reference Data Management Strategy

charlie.mott — Mon, 15 Mar 2010 00:32:49 GMT

Article Source: http://geekswithblogs.net/charliemott

This article describes an approach to the management of cross reference data for BizTalk. Some articles about the BizTalk Cross Referencing features can be found here:

Options

Current options to managing this data include:

Maintaining xml files in the format that can be used by the out-of-the-box BTSXRefImport.exe utility.
Use of user interfaces that have been developed to manage this data:
- BizTalk Cross Referencing Tool
- XRef XML Creation Tool

However, there are the following issues with the above options:

The 'BizTalk Cross Referencing Tool' requires a separate database to manage. The 'XRef XML Creation' tool has no means of persisting the data.
The 'BizTalk Cross Referencing tool' generates integers in the common id field. I prefer to use a string (e.g. acme.petshop.country.uk). This is more readable. (see naming conventions below).
Both UI tools continue to use BTSXRefImport.exe. This utility replaces all xref data. This can be a problem in continuous integration environments that support multiple target BizTalk groups (even different clients). If you upload the data for one client/group it would destroy the data for another client/group. Yet in TFS, where builds run concurrently, this would break unit tests.

Alternative Approach

In response to these issues, I instead use simple SQL scripts to directly populate the BizTalkMgmtDb xref tables combined with a data namepacing strategy to isolate client data.

Naming Conventions

All data keys use namespace prefixing. The pattern will be <company name>.<biztalk group>.<data type>. The data must follow this pattern to isolate it from other company\group cross-reference data. The naming convention I use is lower casing for all items.

The table below shows some sample data. (Note: this data uses the 'ID' cross-reference tables. The same principles apply for the 'value' cross-referencing tables).

Table.Field	Description	Sample Data
xref_AppType.appType	Application Types	acme.petshop.erp acme.petshop.portal acme.petshop.assetmanagement acme.petshop.ordermanagement
xref_AppInstance.appInstance	Application Instances (each will have a corresponding application type).	acme.petshop.dynamics.ax acme.petshop.dynamics.crm acme.petshop.sharepoint acme.petshop.maximo acme.petshop.oms
xref_IDXRef.idXRef	Holds the cross reference data types.	acme.petshop.vatcode acme.petshop.country
xref_IDXRefData.CommonID	Holds each cross reference type value used by the canonical schemas.	acme.petshop.vatcode.exmpt acme.petshop.vatcode.std acme.petshop.country.usa acme.petshop.country.uk
xref_IDXRefData.AppID	This holds the value for each application instance and each xref type.	UK GB 44

SQL Scripts

The data to be stored in the BizTalkMgmtDb xref tables will be managed by SQL scripts stored in a database project in the visual studio solution.

File(s)	Description
Build.cmd	A sqlcmd script to deploy data by running the SQL scripts below. (This can be run as part of the MSBuild process).
acme.petshop.purgexref.sql	SQL script to clear acme.petshop.* data from the xref tables. As such, this will not impact data for any other client/group.
acme.petshop.applicationInstances.sql	SQL script to insert application type and application instance data.
acme.petshop.vatcode.sql acme.petshop.country.sql etc ...	There will be a separate SQL script to insert each cross-reference data type and application specific values for these types.

XSD Validation from BizTalk Orchestrations

charlie.mott — Tue, 16 Feb 2010 16:49:11 GMT

Article Source: http://geekswithblogs.net/charliemott

There are various blog articles that give sample .NET code that can be used to validate a message against a schema from a BizTalk orchestration. These include: msdn, haloscan.com, biztalkgurus.com, eggheadcafe.com and Sujan Turlapaty.

Many of these blogs have subsequent comments about problems. Under high loads, I too began to see these classes return “false positives” in my test environment. (i.e. An XmlSchemaValidationException is being thrown against valid messages. When the messages are re-submitted individually, they pass validation without problems.)

Doing a bit more research into this, I found this Microsoft XML Team's WebLog article about the lack of thread safety of the XmlSchema and XmlSchemaSet classes.

This article also states that "for some reason, this "breaks" more on 64-bit machines, but it's unsafe on all architectures". This would explain why I only began to see the problem when we started testing on 64-bit servers. I never experienced the problems on my 32-bit development or build machines. Nor could I break the code in a unit test using Roy Osherove's ThreadTester. So, another lesson learnt - always develop on the same OS type as the target machines.

To circumvent this problem, I am now calling a validation pipeline from the orchestrations. (I am using a pipeline with Saravana Kumar's Extended XML Validation Pipeline Component so that exceptions report all the validation errors, not just the first). This is working well. However, it does mean that I have had to set the orchestrations to run as long-running transactions.

Update (19/02/2010): Both the out-of-the-box validation pipeline component and Saravana Kumar's Extended XML Validation Pipeline Component, use the deprecated XmlSchemaCollection class. This class is more thread safe than the replacement XmlSchemaSet class. As such, rather than the above solution of calling a pipeline from an orchestration, another solution would be to modify the validation method to use the XmlSchemaCollection instead of the XmlSchemaSet class.

BizTalk Map Documenter

charlie.mott — Thu, 23 Jul 2009 20:34:17 GMT

Article Source: http://geekswithblogs.net/charliemott

The "HTML Generator StyleSheet for BizTalk Maps" originally posted by Steve Hart is now on CodePlex.

http://biztalkmapdoc.codeplex.com

I have uploaded a new version that includes the following updates (not all of this was developed by me):

output of label data.
output of constants used.
updated table layout.
separate tables for each BizTalk page.

I've also included details of how this can be run in an MSBuild process.

Stub of Dynamics AX for BizTalk Development and BizUnit Testing

charlie.mott — Wed, 13 May 2009 21:26:40 GMT

Article Source: http://geekswithblogs.net/charliemott

This article describes our approach to testing BizTalk integration with Dynamics AX 2009. It builds on the "Alternative Bindings" approach as described by Mike Stephenson.

Alternative Bindings

We are communicating asynchronous with Dynamics. As such, in our developer / unit test bindings, we have replaced use of the AIF Adapter with the MSMQ Adapter. If you are communicating synchronously, you could use the WCF Adapter.

Mimic the Dynamics AIF Adapter

We also need to mimic the actions of the AIF Adapter. To do this, we have used 2 test pipelines and a test schema in our developer bindings:

We created a test implementation of the AX envelope schema (AxEnvelope.xsd). This has the same namespace and structure as the actual AX schema. The difference is that our test schema has all the Header fields as promoted properties.

On the send side, our send ports use our MimicDynammicsAifAdapterSend pipeline. This has an XML Assembler component. This wraps the outbound messages in the AxEnvelope envelope. With property demotion, the header fields are set from the context properties (as set in our orchestrations). This enables us to test these values in our BizUnit validation steps.

On the receive side, our MimicDynamicsAifAdapterReceive pipeline has an XML Disassembler that strips off the envelope. The header fields are prompted to the message context properties. In particular, the MessageId envelope header field is required to be added to the message context so the response messages can be correlated to the message sent to Dynamics. The dissassember references the schemas: DynamicsAx5.Fault, DynamicsAx5.EntityKeyList, DynamicsAx5.EntityKey and all other message types we are expecting from Dynamics.

Stub Dynamics

In order to mimic Dynamics sending a response to messages we send to Dynamics, we also built a custom BizUnit test step (AxSendResponseStep).

The response message can be a valid response (DynamicsAx5.EntityKeyList) or an invalid response (DynamicsAx5.Fault) as specified in the step parameters. In the Dynamics response message, the MessageId is replaced with the same MessageId in the received message. This ensures the Dynamics response messages can be correlated by BizTalk orchestrations.

Update: 03/04/2011 - This approach only applies to integration with AX4 and AX5. These versions provide the AIF Adapter. In AX6 (AX2012), services are exposed as WCF services. As such, regular approaches to stubbing out WCF services can be used.