BizTalk: Deployment Hell & BizTalk Deployment Framework

Is there something special in the BizTalk Server application deployment? Why is it so special?

BizTalk Deployment Hell

For the .NET applications the live is simple. There is an exe file and maybe several additional dll-s. Copy them and this is pretty much all deployment we need.

The BizTalk Server requires the dll-s placed in GAC and registered in the special Management database. Why this is required? Mostly because the BizTalk automatically hosts applications in the cluster and because of the reliability. BizTalk application maybe is not easy in deployment but it also not easy to break. For example, if the application A is related to the application B, BizTalk will prevent us from removing B accidentally.

This Why? question is a big theme and I will not cover it here.

Another factor is the BizTalk has many pieces that require a special treatment in deployment and run-time.

Yet another factor is the BizTalk application integrates the independent applications / systems, that means the application should take care of many configuration parameters, as endpoint addresses, user names and passwords and so on.

As a result the BizTalk application deployment is complex, deployment is error prone, slow, unreliable, requires a good amount of resources.

And look here, is is a savior, the BizTalk Deployment Framework.

The BizTalk Deployment Framework - the Champion

The BizTalk Deployment Framework (BTDF) is an essential tool in arsenal of the BizTalk Server developers and administrators. It solves many problems, it speeds up the development and deployment enormously.

It is a definitely a main face in the BizTalk Server Hall of Fame. 

BTDF was created by Scott Colestock and Thomas Abraham.

It is an open source project despite the fact, that BTDF is more powerful, reliable and thorough than the most of commercial BizTalk Server third-party tools. I think it is fair to donate several dollars to those incredible guys on the Codeplex. Just think abut days and months we save in our projects, in our private life.

BTDF is an integration tool and it is created by guys with pure integration mindset. It integrates the whole bunch of open-source products in one, beautiful BTDF. Below I copy the "Contributors and Acknowledgements" topic from the BTDF Help:

Thanks also to:

  • Tim Rayburn for contributing a bug fix and unit tests for the ElementTunnel tool
  • Giulio Van for contributing ideas and bug fixes.
  • The hundreds of active users across the world who have promoted the Deployment Framework, reported bugs, offered suggestions and taken the time to help other users! …”

And how exactly BTDF saves our life?

BTDF is installed and tuned up. Now it is time to deploy an application.

DeploymentMenu.Deploy

The deployment was successful and I’ve got the long output, what was done in this deployment.

Here is a log. I've marked my comments in the deployment log with “+++”. The full redeployment lasted 2 minutes.

If I perform the same amount of tasks manually, I would spent at last 3-4 minutes concentrated on this long list of deployment tasks. The probability of missing some steps would be quite high. With BTDF I start deployment and free to do another tasks.

So my gain is 3-4 minutes on each deployment. Risk of an error is zero, everything is automated.

There is one more psychological problem with manual deployment. It is complicated and it requires the full concentration. As a developer I am concentrated on the application logic and this manual deployment task breaks my concentration. Each time after deployment I have to concentrate on my application logic again. And here is where the development performance goes down to the hell.

There are other helpful BTDF commands:

DeploymentMenu.All

  • Restarts all host instances, only those are used in this application; restart IIS, if I deploy web-services, all with the "Bounce BizTalk" command.
  • Terminate all orchestration and messaging instances remained after tests.
  • Install the .NET component assemble into GAC.
  • Include the modified configuration parameters into a binding file.
  • Import this binding file.
  • Update SSO with modified configuration parameters.

All this with one click.

You do not have to use the slow BizTalk Administrative Console to do those things anymore.

Check my comments in the BTDF deployment log. Several automated tasks are like blessing! We do not create the drop folders anymore, do not assign permissions to those folders. BTDF makes this automatically.

We do not care about undeployment and deployment order, BTDF makes everything right.

We do not stop and start ports, orchestrations, applications, host instances, IIS. Everything is automated.

Look at the list of the top level parameters in BTDF:

DeploymentProperties

Remember, I told you the BizTalk application deployment is a little bit complicated? All those application components in this list require a little bit different treatment in deployment. In a simple application we do not use all those components, a typical application uses maybe 1/3 or 1/2 of them, but you have an idea.

How to tune up BTDF?

One thing I love in BTDF is the Help. It is the exemplary, ideal, flawless help.

If you never tried BTDF, there is a detailed description of all parts, all tasks. Moreover there is the most unusual part, the discussions of the BTDF principles and the BizTalk deployment processes. I got there more knowledge about the BizTalk Server deployment than in the Microsoft BizTalk Server Help - official documentation.

BTDF Help helps if you are a new user or if you use BTDF several years in row. Descriptions are clear, they are not stupid and arranged in clear hierarchy. BTDF Help is one of the best. You are never lost.

Of course there is a detailed tutorial in Help and the sample applications.

OK Now we have to start with tuning.

Typical BTDF workflow for setting up the deployment of a BizTalk application:

  1. Creating a BTDF project.
  2. Setting a deployment project.
  3. Creating an Excel table with configuration parameters
  4. Setting a binding file

All configuration parameters are managed inside an Excel table:

Forget about managing different binding files for each environment. Everything is inside one Excel table. BTDF will pass parameters from this table into binding file and other configuration stores.

Excel helps when we compound parameters from several sources. For example we keep all file port folders under one root. The folder structure below this root is the same in all environments, only the root itself is different. So there is a parameter "RootFolder" and we use it as a part of the full folder paths for all file port folders. For example, we have a "GLD_Samples_BTDFApp_Port1_File_Path" parameter which defined in the cell with formula: = "RootFolder" + ""GLD_Samples_Folder" + "BTDFApp_Folder" + "Order\\*.xml" (of course in the Excel cell it would be something like = C22 & C34 & C345 & "Order\\*.xml"). If we modify the RootFolder path, all related folder paths will be modified automatically.

OK We work hard setting up the application development. Now surprise, we have to create a new environment. The best part of BTDF configuration fiesta is there: all configuration parameters for ALL ENVIRONMENTS for an application are here just in one table. (If you are tenacious enough you could keep a single table for ALL applications, but you have to ask me how.)

Settings for different environments: In our Excel table we copy-past a column of one of the existed environment to a new column for a new environment. Then we modify the values in this new column. And again, this is an Excel table, and if we define the RootFolder variable for a new environment and voila, all file port paths for this environment are modified.

Now we have to pass those configuration parameters to the binding files, right?

We replace all values in the binding file which are different for different environments, like the Host names, NTGroupName, Addresses, transport parameters like connection strings, etc. We replace these values with the variables, like this:BindingFileParameters

Now we save this binding file with the PortBindingsMaster.xml name.

That is pretty much everything we need. Now execute the Deploy BizTalk Application command and application is deployed.

Deployment into a Production environment is different. It is limited in the installed tools and we have to do additional installation steps. BTDF creates an msi and command file. This msi includes all additional pieces we need to install. We don’t have manually add resources to the BizTalk application in the Administrative Console anymore.

Conclusion: BizTalk Deployment Framework is a mandatory tool in the BizTalk development. If you are a BizTalk developer you must use it, you must know it.

Complex XML schemas. How to simplify?

[Sample code is here: ]


The XML Schemas are used for two main tasks: 

  • for processing XML documents (for the XML document validation and for the XML document transformation); 
  • for defining the domain specific standards.

XML Schemas and Domain Standards

Let's talk about the domain standards. EDI, RosettaNet, NIEM, ebXML, Global Justice XML Data Model, SWIFT, OpenTrave, Maritime Data Standards, HIPAA, HL7, etc. If we look at those standards, we see that schemas embrace the domain knowledge in form which can be formally and officially validated. The XML Schemas are very helpful for such tasks.

Compare standards which defined in form of the XML schemas and in form of the documents. It is almost impossible to verify if the data satisfy the standard or not if we use the text document where this standard is defined. And it is possible to validate it and validate it automatically, if we use the XML Schemas.

The domain specialists use XML Schemas to define standards in unambiguous form, in machine verifiable format.

Those schemas tend to be large, huge and very detailed. And it is for very good reasons.

But if we start to use XML Schemas for the first task, for processing XML documents in our programs, we need something different, we need the small schemas. In the system integration we need small schemas.

We don't need the abundance of HIPPA schemas in most applications. We only need a small portion of schema to validate or transform the significant for this application part of schema.

We upload the megabyte size schemas, we perform mapping for these huge schemas, and it lasts for eternity and it consumes huge amount of CPU and memory.

For the most integration projects we don’t want to validate data to satisfy the standard. We want to transfer data between systems as fast as possible with minimal development effort.

How to work with those wealthy schemas? How to do our integration fast at run-time and in development?

First we have to decide, does our application require the whole schemas or not?

If the answer is "No" we could read further to solution.

How to Simplify?

Solution is to simplify the schema. Cut out all unused parts of schema.

The first step in our simplification is to decide which parts of original schema we want to transfer further, want to map to another schema. We keep these parts unchanged and we simplify all other unnecessary schema parts.

The second step is to research if the target integrated system perform the data validation of the input data or not. Good system usually validates input data. Validation includes the data format validation (is this field integer, date type or does it match a regex expression?), the data range (is this string too long or is this integer too big?), the right encoding (is this value belong to the code table?), etc.

If the target system performs this validation, it doesn't make sense to us perform the same data validation on the integration layer. We just pass the data without any validation to the target system. Let this system validate data and decide what to do with errors: send errors back to the source system or try to repair or something else. Actually it is not good architecture, if an intermediary (our integration system) is trying to do such validations and decisions. It means spreading the business logic between systems where target system delegates the data validation logic to intermediary. The integration system deals with data validation only if it needed.

Example: HIPAA Schema Simplification in the BizTalk Server

Now let's be more technical. The next example is implemented in the BizTalk Server and the HIPAA schemas, but you can use the same principles with other systems and standards.

The first step in the schema simplification is the structural modification. It is pretty simple. We replace the unused schema parts with <any> tags [http://www.w3.org/TR/xmlschema-0/#any]. If we are still want to map this schema part but without any details, we can use the Mass Copy functoid.

The second part of the schema simplification is the type simplification.

For the HIPAA schemas I use these regex replacements:

Open your schema with XML (Text) Editor mode:

image

Click Ctrl-Shift-H (Find and Replace in Files) and check “Use Regular Expressions” option:

image

Make two replacements:

  • type="X12_.*"  --> type="xs:string"
  • <xs:restriction base="X12_.*">.*\n.*\n.*\n.*</xs:restriction>  --> <xs:restriction base="xs:string"/>

Save and close.

Open schema again with Schema Editor, make any small change and undo it. Editor will recalculate type information and pops up the Clean Up Global Data Types window. Check all types and click OK.

image

This cleans up all unused Global Data Types.

Previously we replaced all those types with “xs:string” type and those types are not used anymore.

It takes 5 minutes for this replacement. What is the result?

image

The modified schema is twice smaller.

image - is the dll size with original schema.

image - is dll size with modified schema.

The assembly for modified schema also cut twice in size.

Result is not bad for 5 min job.

How these simplified schemas change our performance?

All projects with schemas and maps are compiled in Visual Studio notably faster. I like this improvement as a developer.

How about the run-time performance?

I have made a simple proof of concept project to check the performance changes.

Test project

The project compounded of two BizTalk applications and two BizTalk Visual Studio projects. Do not do this in production projects! One Visual Studio solution should keep one and exactly one BizTalk application.

Each project keeps one HIPAA schema, one very simple schema, one “complex” map (HIPAA schema to HIPAA schema), and one simple map (HIPAA schema to the very simple schema).

The first project works with original HIPAA schema and the second project with simplified HIPAA schema.

Build and Deploy one project.

Each BizTalk application compounded of a receive file location and a send file port. The receive location uses the EdiReceive pipeline to convert the text EDI documents into the XML documents. So we need to add a reference to the “BizTalk EDI Application”:

image

After deployment import the binding file which you find in the project folder. Create the In and Out folders and apply necessary permissions to those folders. Change the folder paths in the file locations for your folders.

There is also a UnitTests project with several unit tests. Change folder paths in the test code.

Perform tests.

Then delete the application and deploy second BizTalk project and perform tests again.

Do not deploy both projects side by side.

Performance results:

Note: Before each test start the Backup BizTalk job to clean up a little bit the MessageBox.

image

Tests for 1, 10 and 100 messages did not show visible difference. The difference could be noticeable in my environment in 1000 message and 3K message batch tests. The above table shows the test result for 3K batch tests.

The performance gain is about 10%. It is not breathtaking but anyway it is not so bad for the 5 minutes effort.

Conclusion: The schema type simplification is worth to do if the application expects the sustainable high payloads, the high peak payloads, and everywhere you want to get the best possible performance.

BizTalk: Complex decoding in data transformations

Sometimes we need to make complex decoding in the data transformations. It could happen especially in the big EDI documents as HIPAA.

Let’s start with examples.

In one example we need to decode a field. We have the source codes and the target codes for this field. The number of both codes is small and the mapping is one to one or many to one (1-1, M-1). One of the simplest solution is to create a Decode .NET component. Store the code table as a dictionary and decoding will be fast. We could hard code the code table, if codes are stable, or extract it in cache, reading it from a database or a configuration file.

The next example is on the opposite side of complexity. Here we need to decode several fields. Target codes related to several source codes/values (M-1). It is not the value-to-value decoding but this decoding also includes several if-then-else conditions, which opens a can of worms with 1-M and M-M cardinality.
Moreover the code tables are big and cannot be placed in the memory.

We can implement this decoding with numerous calls to the database to get the target codes and perform these calls inside a map or inside a .NET component. As a result for each document transformation we calls database many times.

But there is another method of implementing this without flooding the database with these calls. I call this method the “SQL Decoding”.

We might remember that SQL operations are the set operations working with relation data directly. The SQL server is very powerful in executing these operations. Set operations are so powerful, we might decode all fields in a single operation. It is possible, but all source values should be in database at this time. All we have to do is to load the whole source document to the SQL data. Hence our method is:

  1. Load the source message to SQL database.
  2. Execute encoding as a SQL operation or series of operations.
  3. Extract the target message back from the SQL.

We can do all structure transformations in maps (XSLT) and perform only decoding in SQL form. Or we also can do some structure transformations in SQL. It is up to us.

The Pros of this implementation are:

  • It is fast and highly optimized.
  • It does not flood database with numerous calls.
  • It nicely utilizes the SQL engine for complex decoding logic.

The Cons are:

  • Steps 1 and 3 can be not simple.

In real life we usually don’t have a clear separation between our scenarios and the intermediate solutions can be handy. As an example we can load and extract not a whole message but only part of it, related to decoding.

Personally, I use the SQL Decoding in the most complex cases where mapping takes more than 2 days of development.

Note:

  • If you are familiar with LINQ, you can avoid steps 1 and 3 and execute set operations directly on the XML document. I personally prefer to use LINQ. But if XML document is really big, SQL approach works better.

Conclusion: If we need a complex decoding/encoding of the Xml document, consider to use the set operations with SQL or LINQ.

BizTalk Server: to be successful

What should be presented in the successful software?
[By Raymond Hettinger from EuroPython 2011]
PiecesOfSuccess

It was told about Python and it is interesting to use the same criteria for the BizTalk Server.

So, how it plays for the BizTalk Server? My score is (from * to *****):

License – **

Commercial Distributions – ***

Zen – **

Community – ****

Third party software – ****

Killer Apps and Success Stories – **

Platforms –

Books – ***

Now in more details:

License

BizTalk license is a typical server license from Microsoft. And it might be the most expensive license between Microsoft products.

Which is questionable.

The free license might be the best license for the BizTalk Server.

Usually it is told the BizTalk license is very cheap comparing the big brothers: IBM WebSphere and Tibco.

But there is one thing, BizTalk Server is not one of the revenue source for Microsoft. It is an enabler, it is a Trojan horse. BizTalk Server breaks holes in the big data centers of big international companies, where mainframe is the king. BizTalk breaks holes and Microsoft SQL Servers and Windows Servers and other Microsoft enterprise software leaks in those holes. And that is a real revenue source.

So maybe, maybe the free BizTalk Server would be the better Trojan horse Smile

Previous score was 3, now it is 2. Customers are more and more aware of power of open source. But recently Microsoft took a step to increase a license price. Customers and community embrace this decision with discontent and confusion.

Commercial Distributions

Microsoft claims 12,000 customers in 2012. Which is a lot for costly software. This distribution base has a big impact on the BizTalk Server. It is pricey to buy BizTalk, to develop and deploy applications. It is much pricier to get rid of it, if you decide there is something better.

Now Obamacare is adding many new customers because of good HL7 and HIPAA support in the BizTalk Server.

The score is 3, good enough now and will be better in near future.

Zen

Zen was 4 in 2004-2006 when BizTalk 2004 was the first Microsoft commercial server software based on .NET. Philosophy in the base of the product was, and still is, beautiful.

Then were years when Microsoft changed favorites and ceased development investments. BizTalk was not a new SQL Server, it was not a new SharePoint, it doesn’t match in revenues.

Best Microsoft people are not anymore in the BizTalk development now.

BizTalk developers and administrators are still working with tools created in 2004-2006. It is not fun anymore. It is not fun. New BizTalk versions added the new adapters, new SQL Server and Visual Studio. Not exciting. Fun is somewhere else. Young developers are not amazed with BizTalk. Old developers are looking to the new stuff: Azure, Spring Integration, Zato.

Richard Seroter blogged about BizTalk Server once onlce in this year, which is ridiculous! OK twice, but still…

So Zen score is only 2.

Community

BizTalk amazingly has a good community support.

As a moderator of the MSDN BizTalk Forum I use the “Report as abuse” button only once a year. Why? Because everyone here are bloody polite and highly professional. You get your answer here in most cases, no need to pay Microsoft for support.

There are a lot of blogs about BizTalk Server, really good blogs.

Microsoft is in good relations with BizTalk community. There was a special BizTalk Server Summit recently, and the next Summit will be in November, 2013. BizTalk team frequently keeps meetings with BizTalk MVPs. (They were renamed to the Microsoft Integration MVPs and joined with Connected Systems/Azure MVPs which is understandable.)

Community score is 4.

Third party software

There are dozens of third party adapters on the market. A lot of open source utilities on Codeplex and MSDN Gallery and everywhere in internet. Some of them just great like the BizTalk Deployment Framework which is free or the BizTalk360 and AIMS for BizTalk which are not free.

Microsoft is not trying to fix this improving APIs inside BizTalk or organizing something like an app store for the BizTalk utilities/components. Codeplex keeps about two hundred of the BizTalk free utilities.

A third party software now is a part of the BizTalk. For example, the EDI subsystem was originally the EDI Accelerator from Covast, the BizTalk PowerShell Provider was an open source project, and the ESB Toolkit was a project started by Neudesic.

The score is solid 4.

Killer Apps and Success Stories

BizTalk has great case studies. For example, as the Case study shows, Allscripts processes 2.75 milion healthcare transactions a day and builds hundreds of new interfaces each year. It is not too hard to find out the successful stories al across different industries. BizTalk especially shines in financial and health care industries.

But Microsoft struggles to persuade public that such stories are the real Killer Apps.

Score is 2.

Platforms

BizTalk Server works only on the Windows Server, .NET, and SQL Server platform. Is it important, that it can work on other OS and databases?

BizTalk Server integrates any other platforms, including IBM mainframes, Oracle, MySQL.

The number of supported platforms is important for the programming languages. It is not important for the integration tool like BizTalk Server. It doesn’t matter which platform it is used to work. It only matters what platforms and systems it can integrated.

So the score is irrelevant here.

Books

You can find tons of book about BizTalk. You can find a lot of fresh books, because BizTalk Server is mature product now and developers and architects got so much experience to share.

What about the BizTalk documentation? It is abundant. But… Creators of the BizTalk did not create or participate in creation of documentation. The top level of documentation, the creator’s point of view, the trades off, the top requirements, and the architecture decisions are not in the BizTalk documentation. A lot of “button by button” instructions could not replace, could not cover this deep hole in the documentation, this blocks learning of the BizTalk.

Score is 3.

Conclusions

All those was a chinwag, a speculation.

The truth is kept secret inside Microsoft.

Only Microsoft knows, is the BizTalk Server a successful software?

BizTalk Integration Development Architecture

You can find some architecture information into the BizTalk documentation. You find out several tutorials and good amount of samples. Almost all of them related to the infrastructure architecture, i.e. how to create high available systems with clusters, how to scale out the BizTalk systems, etc. 
I covered several development aspects of architecture in series of articles:

Copying a new build to all environments

I am doing this task again and again, so maybe this code will be helpful not only for me.

That is a standard routine. I am developing a BizTalk Server applications and use the BizTalk Deployment Framework (BTDF) for all my deployments. When an application is ready for testing and, at the end, for production, the build files have to be deployed. Usually the BizTalk installation has several environments. For example, the environments can be: Development, QA, Staging, Production. Sometime less, sometime more. The best practice is to keep all environments isolated of each other. So each environment keeps its deployment packages separately. That means, in my case, the build files should be copied to all environments.

A good practice is to save the old builds in case of rollback.

The folder structure for the builds looks like this:

image

The Current folder keeps the currently deployed build. The [YYYYMMDD_hh_mm_ss] folders keep the old builds.

What is interesting in this code?

The Copy performs two nested loops. One through the EnvironmentName and the second through the NewBuildToCopy files.

Also the code generated the folder name in [YYYYMMDD_hh_mm_ss] format.

Here is the code:


<!-- Copy a new deployment build to all environments and to a Personal share. 
  Before this rename a Current folder to the [CurrentDateTimeTime] to save an old build. 
  -->
<Target Name="AfterInstaller" AfterTargets="Installer">
  <PropertyGroup>
    <NewBuild>..\Deployment\bin\$(Configuration)</NewBuild>
    <CurrentDateTime>$([System.DateTime]::Now.ToString("yyyyMMdd_hh_mm_ss"))</CurrentDateTime>
    <Shares>\\fileshares.domain.com\Shares\</Shares>
    <SourceCodeShare>\BizTalk\Deployment\$(ProjectName)</SourceCodeShare>
    <PersonalShare>Z:\Projects\BizTalk\GLD\Samples\Deployment\$(ProjectName)</PersonalShare>
  </PropertyGroup>

  <!-- Rename Current shares to the [CurrentDateTime]: -->
  <ItemGroup>
    <EnvironmentName Include="QA;STG;PROD"/>
  </ItemGroup>
  <ItemGroup>
    <CurrentShare Include="$(Shares)%(EnvironmentName.Identity)$(SourceCodeShare)" />
    <CurrentShare Include="$(PersonalShare)" />
  </ItemGroup>

  <Exec Condition="Exists('%(CurrentShare.Identity)\Current')"
         Command='Rename "%(CurrentShare.Identity)\Current" "$(CurrentDateTime)"'/>

  <ItemGroup>
    <NewBuildToCopy Include="$(NewBuild)\**\*.*">
      <Destination>%(CurrentShare.Identity)</Destination>
    </NewBuildToCopy>
  </ItemGroup>

  <!-- Copy the last build to the Current shares: -->
  <Copy Condition="@(NewBuildToCopy) != ''"
        SourceFiles="@(NewBuildToCopy)"
        DestinationFiles="@(NewBuildToCopy->'%(Destination)\Current\%(RecursiveDir)%(Filename)%(Extension)')" />
</Target>


This target could be a part of the Deployment.btdfproj file (which is a file from the BTDF Deployment project). Also you can add it to the BizTalkDeploymentFramework.targets file.

BizTalk: Custom API: Promoted Properties

How to get the Promoted Properties within .NET code? This sample exposes very simple API to access all Promoted Properties, currently deployed into a BizTalk group.

The Best Application Server from Microsoft

-Are you stupid? The BizTalk Server is an Integration Server. It is nothing to do with Application servers.

That’s what you are probably thinking now…


Application Types

Let’s discuss different application types.

Sequential Processing Application

These applications work as single-threaded processes. An application gets the data potions and processes them one after another.
. BatchProcessing

One exemplar of such kind is a file processing application. An application reads a file and processes it, then it reads another file, etc. An application starts with a new file only after finishes with a current file. It works with data in sequence; it processes one data portion then starts with another.

OS works as an application server for such applications. The applications are started by users of when OS is started. Widows Service is such application. OS provides the additional management features like an automatic restart after failure, etc.
image 

 

The Service [Application]

An example of this kind of applications is a web-site. An Application Server example is the Internet Information Server (IIS). A Service Application is constantly waiting a request from clients. A separate service instance is started for each client request.

ShortRunningTransactions

If there are too many requests, the application server places the waiting requests in a queue, so the requests are not over flood the server. For the sequential application only a single application instance is working at the time, for the service application many application instances are working simultaneously, one instance per a client request.

Usually a service does not store the client data to a durable storage. If the service instance is crashed for any reason, a client just repeats a request and a new service instance is created. Reliability is good, if the service instance works for a short period of time. But if a service instance works for hours or days, the possibility of the crash increases and reliability is not good.

The Long-Running Service

In the simplest case a service instance gets a client request, processes it, returns the result data, and that’s it. A little bit more complex case, when a service instance itself requests data from an external service. The life duration on this service instance is unpredictable in this case. We cannot manage the external services, we don’t know when they response. Our service instance can possible wait hours and days, and the Application server can move the service instance snapshot to the durable storage and move it back when it gets the result from an external service. This process is called dehydration-hydration, and it increases the service reliability. Also it frees the computer resources, a big number of dehydrated service instances do not clog the computer.

LongRunningTransactions

Another problem with long-running services is the request correlation.

Everything is simple if a new independent service instance is created for every client request, a service instance process a request, and returns a result to client. But now the service instance itself requests an external service. Imagine hundreds service instances are working simultaneously and all of them send requests to an external service. The external service processes these requests and returns the responses back. As a result we have hundreds service instances and hundreds responses. How to route a response to the right service instance, which requested exactly this response? Each such response must have some information which links this response exactly with the correct service instance. The process of linking is called the correlation. For example, the service instance Id is included in the request, and then this Id is moved into a response and is used for routing this response back, to correlate this response with the right service instance.

The correlation date should be unique, so each response can be correlated with one and only one service instance.

Ideally the application server provides the correlation mechanism.

Another problem with long-running services is the “dead instance cleanup”. For example, a service instance makes a request to an external service and doesn’t get a response back for predefined period of time. The external service could be down, the network could be overloaded, etc.; there could be many reasons in the distributed world. In another example, the service instance is hanged and does not response in any way. It would be cleaver to declare such service instances as “dead” and remove them from memory.

One popular method for the dead instance cleanup is the “heartbeat measure”. A special heartbeat service periodically requests all working service instance and if some instance does not response, cleanup this instance from memory.

The requests to the external services can return errors for different reasons. Sometimes the external service returns legitimated error and we cannot do anything with it. A request is processed with error and it will be an error if we repeat the processing. But most frequently the error is temporary. For example, the network was overloaded, the database was overloaded, the I/O queue is overcrowded, and the external service is busy. In such cases the request could be repeated after a small delay and after several times we get the good response back. The error will be raised, if the request is retried defined number of times.

So we have a list of additional functionality for the application server which works with the long-running services:

  • a durable storage for the service instances
  • a correlation mechanism
  • a system for monitoring and cleanup of the dead instances
  • a retry system of the requests to the external system


And it happens, the Microsoft such application server in its arsenal. It is the BizTalk Server.

The BizTalk database, which is called the Message Box, performs as the reliable data storage. The service instance could dehydrate in this storage and wait here infinite time. Thousand, hundred thousand service instances could wait and the performance of the whole system is not decreased.

The correlation mechanism is a part of the Message Box. Requests are analyzed and routed to the service instance, which is waiting exactly this request, or a request starts a new service instance.

There is a heartbeat and the dead instance cleanup.

There is a retry system which automatically repeats the requests to the external services.

And here is the main feature of the BizTalk Server, its famous reliability. The BizTalk Server processing power simply spreads between several computers and data is stored on the SQL clusters. Deployed applications are working for years without any attention. Power grid could be down, the servers could crash, the network went down, the hard drives could fail. But the applications are not. The BizTalk Server automatically restarted, hundreds or thousands interrupted service instances restore and work as nothing had happened.

Upgrade an Application from the BizTalk Server 2010 to 2013. One error

I was moving the projects from the BizTalk Server 2010 to the BizTalk Server 2013 under the Visual Studio 2012.

I have spent a good chunk of time investigating this error and decided to blog on it to save you this time.

The error happens at deployment time at this command:

BTSTask.exe AddResource -Type:BizTalkAssembly -Source:"..\<ApplicationName>\bin\Debug\<ApplicationName>.dll" -ApplicationName:"<ApplicationName>" -Options:GacOnAdd,GacOnImport,GacOnInstall

or when I Deploy a BizTalk project from the Visual Studio 2012.

error DEPLOY: Access to the path 'C:\Users\…\AppData\Local\Temp\2\BT\PID3656\BizTalkAssembly\c871b7bbc5c2ac36b7da3592c65912d5\<ApplicationName>.dll' is denied.

this error was coupled with another error

error MSB3073: The command "BTSTask.exe AddResource -Type:BizTalkAssembly -Source:"..\<ApplicationName>\bin\Debug\<ApplicationName>.dll" -ApplicationName:"<ApplicationName>" -Options:GacOnAdd,GacOnImport,GacOnInstall" exited with code 1. […\<ApplicationShortName>\Deployment\Deployment.btdfproj]

Do you think it is something with permissions? Access is denied, right?

Nope. Nothing was wrong with permissions.

The problem was in the AssemblyInfo.cs file which is under the Properties folder of the project. For some reason this file was not stored in the source control and was not moved to the upgraded project.

To resolve this error create this file and… That’s it.

[UPDATE] Not so easy, not so easy...

Not any AssemblyInfo.cs is working! Make sure it has a line:

[assembly: Microsoft.XLANGs.BaseTypes.BizTalkAssembly(typeof(Microsoft.BizTalk.XLANGs.BTXEngine.BTXService))]

BizTalk: the Naming Conventions in Examples

See the BizTalk: BizTalk Solution Naming Conventions article.

In small application we do not really care about names. When the number of objects start growing, we start pay attention to the names. Experienced developers working with big projects are recognizable by the carefully crafted names in the developed code.

When a new developer start working in a new team, he/she spent first several hours to read the general level documentation and the naming convention usually is a part of it. Then he/she start to develop and now time should be spent to review the existed code base to become accustomed to the coding standards, and again the naming conventions are the main parts of them.

So the naming conventions document are important but the existed code base is also important. The documentation could be not up to date, but the code always is.

The code example easily could replace the the documentation, that’s why I decided to show the BizTalk naming convention in the real-life examples.

Example description

This BizTalk deployment is compounded from many BizTalk applications. Each of application has 1-10 Visual Studio projects. The applications were developed through the many years by different development teams. Applications integrate several systems. Some systems have just one interface, some systems have many interfaces. In most cases one system interface is integrated with another system interface. Sometimes several systems are integrated with one system interface.

One solution is chosen for this example. Why the GLG.Samples.Name.Shared.Schemas solution was created in real life?

As we know an interface in BizTalk defined by a protocol and one or several XML schemas.  In many cases those schemas are dictated by the integrated system not by the BizTalk application developer. Usually the BizTalk developer uses the Adapter Wizard to generate/import these schemas. These schemas are managed (created, modified) not by BizTalk application developer but by the external system owners. Let’s call these schemas the external schemas.

One interesting aspect of the external schemas is one schema can be used by several BizTalk applications. As we know the schemas for the receive locations should be unique, they cannot be deployed in several assemblies. That means if we want to use the same schema in several projects we should share an assembly with this schema, reference this assembly from all these projects.

In this situation the special attention is paid to these schemas.

All external schemas are placed in a separate Visual Studio solution. The schemas are grouped by the systems. Inside each system the schemas are grouped by the interface names.

This design gives us one more useful feature. Here we always know where to find the external schemas. They are not spread through many projects and application but always placed and can be found in one solution.

The sample provide two versions of the naming conventions. I called them "Long Names" and "Short Names".

Long Names

It is the Solution Explorer view:

Solution.Long

It is the folder view:

Folders.Long

Short Names

It is the Solution Explorer view:

Solution

It is the folder view.

Folders

Comparing

The Short Name convention looks more clean. There are no generic prefixes (GLD.Samples.Names.Shared.Schemas) inside the Project and folder names. If you fight the well-known issue with the TFS limitation on the size of the full file names, it would be a preferable variant.

I personally prefer the Long Name conventions. The project name is equal to the project namespace and the project assembly name. The full project names are better in situations where a project might be included in several Visual Studio solutions or the projects could be frequently shuffled between solutions.

Q&A :

  1. Is it OK to use the names compounded from 4 and more pieces like GLD.Samples.Names.Shared.Schemas?
  2. Do we need the GLD.Samples prefix in the BizTalk application name, in the solution name?
  3. Why we use the _Ops suffix for the SystemA, SystemB, and SystemC projects?

BizTalk Applications

Here is the BizTalk Administration Console view:

BizTalkAdminConsole

As you can see the GLD.Samples.Names.Shared.Schemas solution is deployed as the Names.Shared.Schemas application. The GLD.Samples prefix is removed because all deployed solutions are using this GLD.Sample prefix. If all artifacts has the same name part, this part definitely can be removed. Why this part is not removed from the solution/project name? Because projects (assemblies) are working together with many system, Microsoft and others assemblies in one global name space, in .NET run-time space. All assemblies should be placed in GAC, where GLD.Samples prefix helps us to find out our assembles. This is an answer on the second question. Smile

This is the Short Name convention for the application names.

The Long Name convention is simple:

 the BizTalk Application Name equal the Visual Studio Solution Name

I personally prefer the Long Name convention as a faster way to work, copy/past without any change, and do not guess the full name of the assemblies.

How complex could be the name hierarchy?

For example, we have such name as GLD.Samples.Names.Shared.Schemas.Sap.AV_CREDIT_STATUS_MODIFY. Isn't it too complex?

Not at all. All parts of this name are here for a reason. Each part shows the grouping of an object. Our names unambiguous show us the object hierarchy. Just try to avoid some parts to get something shorter and you would be in trouble. You would create not one but numerous naming conventions and numerous rules how to use them.

Hierarchical naming conventions is the king of the BizTalk projects. These projects are complex enough to create the object grouping hence hierarchy. But they are not too complex to push us in the areas, where the hierarchical grouping is not working well, where keys or tags are working better.

I hope this is an answer for the first question.

Now the answer for the third question. Why we use the _Ops suffix for the SystemA, SystemB, and SystemC projects?

For instance, we have GLD.Samples.Names.Shared.Schemas.SystemA.CreditCheck_Ops project
why it is not just
GLD.Samples.Names.Shared.Schemas.SystemA.CreditCheck ?

The problem is, this project has a schema which has several roots and one of them is the CreditCheck. What is wrong with it, why we cannot use the same name for a part of the project name and for a schema root?

The XML schema is serialized into the .NET classes, and a schema root serializes into one such class. So we will get a class with full qualified name as GLD.Samples.Names.Shared.Schemas.SystemA.CreditCheck.ChreditCheck. And while the project is building, we got an error like this:

Error.SymbolXIsAlreadyDefined

Description would be a little bit different from the picture about: “symbol ‘GLD.Samples.Names.Shared.Schemas.SystemA.CreditCheck’ is already defined; the first definition is in assembly…”.

For some reason the .NET builder parses the names sometimes left-to-right, sometimes right-to-left and this creates this kind of errors.

To avoid this kind of error we have to avoid creating .NET classes with name equal one of the namespace part. The "_Opr" suffix is used for this purpose.

This is an implementation detail, but we have to take care of it in the naming conventions.

Conclusion

There is no single naming convention for all and every situation. Be reasonable and do not create an over engineered conventions. They are not the goal in development but the helpers.

Quiz for readers Smile

  • Propose your improvements to these naming conventions.
  • What are the Pros and Cons of placing the external schemas in a separate solution?
  • This solution could be very big. Would be this a problem?

BizTalk: Internals: Schema Uniqueness Rule

A source code could be downloaded from here.

Sometimes in BizTalk the artifact visibility can be global. We place the artifacts in different assemblies and it doesn't change visibility. We place the artifacts in different BizTalk applications and it doesn't change visibility. 
Let’s talk about the schemas. In my previous post BizTalk: Internals: namespaces I've shown that schemas have additional parameter, the Xml [target] namespace. Why it is so important?
The BizTalk receives messages on the Receive Locations. Theoretically messages are just the byte arrays. But the BizTalk frequently has to do additional tasks like changing the message format from the one system format to another. That means the message should be received in an known format. Without knowing the format the BizTalk cannot make a transformation. The BizTalk message internal format is an Xml or a byte array. Here I am talking about the messages in the Xml format.
So the Xml messages are received by the Receive port as the byte arrays and should be parsed into Xml format. The XMLReceive pipeline makes this transformation. The first thing the XMLReceive makes, it searches a Xml namespace and a root node. Those two parameters create a MessageType parameter with is promoted as a context parameter.
As you can see the BizTalk uses the MessageType to find out the schema for this message. This Xml schema is used to parse the whole message into Xml format.
So the first step is to find a Xml namespace and a root node inside the byte array regardless of anything else, the second step is to find out the Xml schema, then this schema is used to parse the whole message into Xml format.
Now we are discussing the second step, how the BizTalk is searching the right schema.
The BizTalk searches through the whole schema list regardless of the application boundaries. Each schema belongs to one of the BizTalk application, but BizTalk ignores this. The global schema list is stored inside the BizTalk management database. When we deploy a project with a schema, the schema is registered inside this database.
When an inbound  message is received and processed by the XMLReceive pipeline, the BizTalk extracts a MessageType (a namespace + a root node), and searches for this MessageType in the management database. An error is raised if the MessageType is not found. The error is raised if more than one schemas with this MessageType is found.

An important note is the schema uniqueness rule is not verified at the deployment time.
I have created several samples to show how it works. These samples are bare-bone projects, focused on the “schema uniqueness” rule.

Identical schemas in the one BizTalk application

The application includes two projects. Each project is compounded from one schema.

OneApp.VSSolution

Both schemas have the same structure and the same Xml namespaces. Schema1

The only difference is the .NET namespaces. Each schema is placed in different assembly. OneApp.DeployedSchemas.

I have created one receive port with one File receive location and XMLReceive pipeline. A send port works as a subscriber to this receive port.

Nothing else is in this application. Two identical schemas in different assemblies and receive and send ports.

I dropped a test Xml file in the receive folder. What do you think is happened? You are right, I've got an error and two related events, 5753 and 5719. OneApp.EventsOneApp.Events.5753OneApp.Events.5719

It is a famous "Cannot locate document specification because multiple schemas matched the message type ###" error. :)

Conclusion: schemas of the receive messages should be unique across all BizTalk application.

Identical schemas in the different BizTalk applications

Now two the same projects are in different BizTalk applications. (As you can see I have broken the naming convention, which forces to place the BizTalk applications in the different Visual Studio solutions. In real life there would be two Visual Studio solutions. Sorry for that. Smile)

VSSolutionFirstApp.Schema1SecondApp.Schema1

The ports are the same as they are in the first sample.

When I dropped a test file I have got the same error as in the first sample. So the schemas are now placed in different applications but it doesn't change the schema visibility. The schema from one application is visible from another application.

Conclusion: schemas of the receive messages should be unique in different BizTalk applications. In other words they should be unique across the whole BizTalk Server Group.

Identical schemas imported by different schemas

Now let’s ask a question. Is the “schema uniqueness” rule relating to the imported schemas?

Here is a real-life example. We are consuming the WCF web-service metadata, and it creates the schema set where we can see the Microsoft Serialization schema. It is the same schema almost in each and every WCF service. Do we have to take care of this schema if we consume several WCF services? Do we have to “fight” this schema, extract it into Shared project, etc.?

This sample is the same as a previous sample with two additional schemas: Importing.xsd and Imported.xsd.
VSSolution 

Both Importing schemas in two applications are different. They have different Xml namespaces.
ImportingSchema

But both Imorting schemas are identical.ImportedSchema

The ports are the same as they were in the first sample.

When I dropped a test file there was no error.

So the “root” schema namespaces, the namespaces which define the MessageType were different, but the namespaces of the imported schemas were the same and this the XMLReceive pipeline could successfully recognize the MessageType of the message.

Conclusion: the “schema uniqueness” rule works only for the schema which define a MessgeType of the received message and doesn't work for the imported schemas.

Notes:

  • Two parameters of Imported schema are changed from default values. It forces to include the namespace prefixes of the Imported schema into the Xml documents. Those parameters are the AttributeFormDefault and ElementFormDefault.

    ImportedQualifiedParam

    A message with these parameter values equal “Qualified”:Xml.Qualified

    A message with these parameter values equal “Default”:Xml.Unqualified
  • We could mention the “typeOnlySchema” Root Name for the Imported schema. Actually this schema doesn't have any root, any elements, it includes only types.
    TypeOnlySchema

  • Note: The samples above are about the XMLReceive pipeline, but the same  “schema uniqueness” rule works also for the XMLTransmit send pipeline.

“Schema Uniqueness” Rule

The schemas used for the MessageType should be unique within the BizTalk Service group if these schemas are used in the XMLReceive and XMLTransmit pipelines.

Is there a reason for the “Schema Uniqueness” rule?

Is there a good reason for this rule?

I would prefer if the schema visibility is limited inside the BizTalk application. It would extremely simplify the schema and service versioning.

Let’s take an example when our partner added a new version of the service. The service schemas itself are different in both services (several nodes were removed, several nodes were added), but the schema namespaces and roots are the same in both services. The new and old services work side-by-side and we have to consume both service versions. If we could limit the schema visibility inside the application boundaries, creating the BizTalk applications for consuming both service versions would be simple. Now it is cumbersome. I am sure, each experienced BizTalk developer can remember several real-life examples, when this “schema uniqueness” rule was a reason when several hours project was instead implemented in several days.

Looks like a reason for this “schema uniqueness” rule is simple. The BizTalk application was introduced only in the BizTalk 2006. In this time Microsoft decided do not change the XMLReceive and XMLTransmit pipelines to embrace new application conception and since then this feature was not on the priority list in the BizTalk development.

BizTalk: Internals: the Partner Direct Ports and the Orchestration Chains

Partner Direct Port is one of the BizTalk hidden gems. It opens simple ways to the several messaging patterns.

This article based on the Kevin Lam’s blog article. The article is pretty detailed but it still leaves several unclear pieces. So I have created a sample and will show how it works from different perspectives.

Requirements

We should create an orchestration chain where the messages should be routed from the first stage to the second stage. The messages should not be modified. All messages has the same message type.

Common artifacts

Source code can be downloaded here.

It is interesting but all orchestrations use only one port type.Schema1PortType

It is possible because all ports are one-way ports and use only one operation.

I have added a B orchestration. It helps to test the sample, showing all test messages in channel.

Orch.B

The Receive shape Filter is empty. A Receive Port (R_Shema1Direct) is a plain Direct Port. As you can see, a subscriptionB.Subscription expression of this direct port has only one part, the MessageType for our test schema: A Filer is empty but, as you know, a link from the Receive shape to the Port creates this MessageType expression.

I use only one Physical Receive File port to send a message to all processes.

Each orchestration outputs a Trace.WriteLine(“<Orchestration Name>”).

Forward Binding

This sample has three orchestrations: A_1, A_21 and A_22.

A_1 is a sender, A_21 and A_22 are receivers.

Here is a subscription of the A_1 orchestration:A_1.Subscription

It has two parts

  • A MessageType. The same was for the B orchestration.
  • A ReceivePortID. There was no such parameter for the B orchestration. It was created because I have bound the orchestration port with Physical Receive File port. This binding means the PortID parameter is added to the subscription.

All

How to set up the ports?

  1. All ports involved in the message exchange should be the same port type. It forces us to use the same operation and the same message type for the bound ports.
  2. This step as absolutely contra-intuitive. Ninja We have to choose a Partner Orchestration parameter for the sending orchestration, A_1. The first strange thing is it is not a partner orchestration we have to choose but an orchestration port. But the most strange thing is we have to choose exactly this orchestration and exactly this port.It is not a port from the partner, receive orchestrations, A_21 or A_22, but it is A_1 orchestration and S_SentFromA_1 port. Surprised smile
  3. Now we have to choose a Partner Orchestration parameter for the received orchestrations, A_21 and A_22. Nothing strange is here except a parameter name. We choose the port of the sender, A_1 orchestration and S_SentFromA_1 port.

As you can see the Partner Orchestration parameter for the sender and receiver orchestrations is the same.

Testing

I dropped a test file in a file folder. There we go: DebugView.Forward

  1. A dropped file was received by B and by A_1
  2. A_1 sent a message forward.
  3. A message was received by B, A_21, A_22

Let’s look at a context of a message sent by A_1 on the second step:

  • A MessageType part. It is quite expected.
  • A PartnerService, a ParnerPort, an Operation. All those parameters were set up in the Partner Orchestration parameter on both bound A-1.MessageSent.Contextports.

 

 

Now let’s see a subscription of the A_21 and A_22 orchestrations. A_2x.Subscription

Now it makes sense. That’s why we have chosen such a strange value for the Partner Orchestration parameter of the sending orchestration.

Inverse Binding

This sample has three orchestrations: A_11, A_12 and A_2.

A_11 and A_12 are senders, A_2 is receiver.

All

How to set up the ports?

  1. All ports involved in the message exchange should be the same port type. It forces us to use the same operation and the same message type for the bound ports.
  2. This step as absolutely contra-intuitive. Ninja We have to choose a Partner Orchestration parameter for a receiving orchestration, A_2. The first strange thing is it is not a partner orchestration we have to choose but an orchestration port. But the most strange thing is we have to choose exactly this orchestration and exactly this port.It is not a port from the partner, sent orchestrations, A_11 or A_12, but it is A_2 orchestration and R_SentToA_2 port. Surprised smile
  3. Now we have to choose a Partner Orchestration parameter for the sending orchestrations, A_11 and A_12. Nothing strange is here except a parameter name. We choose the port of the sender, A_2 orchestration and R_SentToA_2 port.

Testing

    I dropped a test file in a file folder. There we go: DebugView.Inverse

    1. A dropped file was received by B, A_11 and by A_12
    2. A_11 and A_12 sent two messages forward.
    3. The messages were received by B, A_2

Let’s see what was a context of a message sent by A_1 on the second step:

  • A MessageType part. It is quite expected.
  • A PartnerService, a ParnerPort, an Operation. All those parameters were set up in the Partner Orchestration parameter on both bound ports.A-1x.MessageSent.Context

Here is a subscription of the A_2 orchestration.

A-2.Subscription

Models

I had a hard time trying to explain the Partner Direct Ports in simple terms. I have finished with this model:

Forward Binding

Publisher/Sender doesn't know Subscribers/Receivers. Subscribers/Receivers know a Publisher/Sender.
1 Publisher –> M Subscribers

Inverse Binding

Publishers/Senders know a Subscriber/Receiver. Subscriber/Receiver doesn't know Publishers/Senders.
M Publishers –> 1 Subscriber

Notes

 

Orchestration chain

It’s worth to note, the Partner Direct Port Binding creates a chain opened from one side and closed from another.

The Forward Binding: A new Receiver can be added at run-time. The Sender can not be changed without design-time changes in Receivers.

The Inverse Binding: A new Sender can be added at run-time. The Receiver can not be changed without design-time changes in Senders.

BizTalk: Internals: Namespaces

BizTalk team did a good job hiding the .NET guts from developers. Developers are working with editors and hardly with .NET code. The Orchestration editor, the Mapper, the Schema editor, the Pipeline editor, all these editors hide what is going on with artifacts created and deployed.

Working with the BizTalk artifacts year after year brings us some knowledge which could help to understand more about the .NET guts.

I would like to highlight the .NET namespaces. What they are, how they influence our everyday tasks in the BizTalk application development.

What is it?

Most of the BizTalk artifacts are compiled into the NET classes. Not all of them… but I will show you later. Classes are placed inside the namespaces. I will not describe here why we need namespaces and what is it. I assume you all know about it more then me.

Here I would like to emphasize that almost each BizTalk artifact is implemented as a .NET class within a .NET namespace.

Where to see the namespaces in development?

The namespaces are inconsistently spread across the artifact parameters.

Let’s start with namespace placement in development. Then we go with namespaces in deployment and operations.

I am using pictures from the BizTalk Server 2013 Beta and the Visual Studio 2012 but there was no changes regarding the namespaces starting from the BizTalk 2006.

Default namespace

image

When a new BizTalk project is created, the default namespace is set up the same as a name of a project. This namespace would be used for all new BizTalk artifacts added to this project.

Orchestrations

When we select a green or a red markers (the Begin and End orchestration shapes) we will see the orchestration Properties window. We also can click anywhere on the space between Port Surfaces to see this window.

image

 image

Schemas

The only way to see the NET namespace for map is selecting the schema file name into the Solution Explorer.

image

image

Notes:

  • We can also see the Type Name parameter. It is a name of the correspondent .NET class.
  • We can also see the Fully Qualified Name parameter.
  • We cannot see the schema namespace when selecting any node on the schema editor surface. Only selecting a schema file name gives us a namespace parameter.
  • If we select a <Schema> node we can get the Target Namespace parameter of the schema.
    image
    image

    This is NOT the .NET namespace! It is an XML namespace. See this XML namespace inside the XML schema, it is shown as a special targetNamespace attributeimage

    Here this XML namespace appears inside the XML document itself. It is shown as a special xmlns attribute.
    image

     

Maps

It is similar to the schemas. The only way to see the NET namespace for map is selecting a map file name into the Solution Explorer.

image

image

Pipelines

It is similar to the schemas. The only way to see the NET namespace for pipeline is selecting a pipeline file name into the Solution Explorer.
imagez

image

Ports, Policies and Tracking Profiles

The Send and Receive Ports, the Policies and the BAM Tracking Profiles do not create the .NET classes and they do not have the associated .NET namespaces.

How to copy artifacts?

Since the new versions of the BizTalk Server are going to production I am spending more and more time redesigning and refactoring the BizTalk applications. It is good to know how the refactoring process copes with the .NET namespaces. Let see what is going on with the namespaces when we copy the artifacts from one project to another.

Here is an example: I am going to group the artifacts under the project folders. So, I have created a Group folder, have run the Add / Existing Item.. command and have chosen all artifacts in the project root. The artifact copies were created in the Group folder:

image

What was happened with the namespaces of the artifacts?

image

As you can see, the folder name, the “Group”, was added to the namespace. It is great! When I added a folder, I have added one more level in the name hierarchy and the namespace change just reflexes this hierarchy change.  The same namespace adjustment happens when we copy the BizTalk artifacts between the projects.

But there is an issue with the namespace of an orchestration. It was not changed. The namespaces of the schemas, maps, pipelines are changed but not the orchestration namespace. I have to change the orchestration namespace manually.

Now another example: I am creating a new Project folder and moving the artifacts there from the project root by drag and drop. We will mention the artifact namespaces are not changed.

Another example: I am copying the artifacts from the project root by (drag and drop) + Ctrl. We will mention the artifact namespaces are changed. It works exactly as it was with the Add / Existing Item.. command.

Conclusion:

  • The namespace parameter is put inconsistently in different places for different artifacts
  • Moving artifacts changes the namespaces of the schemas, maps, pipelines but not the orchestrations.

BizTalk: mapping with Xslt

BizTalk Map Editor (Mapper) is a good editor, especially in the last 2010 version of the BizTalk. But sometimes it still cannot do the tasks easily. It is time for the Xslt code, it is time to remember that the maps are executed by the Xslt engine. 

Right-click the Mapper Grid (a field between the source and target schemas) and choose Properties /Custom XSLT Path.  Input here a file name for the file with Xslt code. Only this code will be executed now, forget the picture in the Mapper, all those links and functoids. 

Let’s see the real-life example. It is the typical example of the "schema flattening". We have a target record and we have to create it from different unrelated source records.

As we can see, there are two source Addresses. One is on the top level and the second one is inside the Member_Address record with MaxOccurs=* .

A target address is placed inside the Locator record with MaxOccurs=*.

Map.Pict

The requirement is to map all source addresses to one target address.

The source Xml document looks like:

Source.Xml

The result Xml should be like this:

Output.Xml

Try to make up this mapping with the Mapper and you would spent good amount of time and the result map would be tricky.

If we use the Xslt code, the mapping will be simple and unambiguous, like this:

Xslt

There are two identical pieces creating the Locator records. 

The first one is started with <xsl: if test="s2:Address_1". It maps the upper level address to the Locator.

The second one is started with <xsl: for-each select="s2:Member_Address/s2:Member_Address". It also maps the addresses nested inside the /../Member_Address/Member_Address record to the Locator.

Simple, clear.

RabbitMQ Server: Error: Host name in Upper-case

This is a small thing, but very annoying.

I didn’t find it in Internet, so decided to blog it here.

My environment: Windows Server 2008 R2 x64; RabbitMQ 2.7.1.

I installed the RabbitMQ Server and tried to interact with it using the rabbitmqctl.bat script. This script requires to use the host name:

>rabbitmqctl –n rabbit@Win2K8R2 status

I was getting the error again and again:

“Error: unable to connect to node rabbit@Win2K8R2: nodedown”

RabbitServer.Status

 

I used the exact string provided me by script (see above in red) -

- nodes and their ports on Win2K8R2:…

and the error text was -

Status of node rabbit@Win2K8R2

When I changed the host name to upper-case, error was gone.

          >rabbitmqctl –n rabbit@WIN2K8R2 status

Conclusion:

   Use only upper-case letters in the host name in rabbitmqctl.bat script!

BizTalk and RabbitMQ

The same article in the TechNet.

If you are working with queues in BizTalk Server, the most possible it is the MSMQ (in my experience).

MSMQ is an old man of the Microsoft technology stack. It was created when there were no good standards for messaging. Now MSMQ is wrapped partly in the .NET System.Messaging namespace. It is just a small facelift. MSMQ is still a proprietary technology without well-defined messaging protocol. It means, you cannot use the MSMQ messaging protocol without MSMQ itself.

Now we see vortex of new messaging standards and technologies. One on the top is the AMQP standard and one of the bold implementation of AMQP is the RabbitMQ.

It is one more protocol, one more messaging system which, for sure, can be integrated with BizTalk Server.

Here I will implement the standard queue messaging: sending and receiving messages from BizTalk down through the RabbitMQ queue. I am using the WCF-Custom LOB Adapter (provided from BizTalk side) and the WCF RabbitMQ Binding (provided from RabbitMQ side) to link two systems.

Installation and preparation

Assuming the BizTalk Server and the Visual Studio 2010 are also installed.

Installing RabbitMQ Service

  1. Go to the RabbitMQ site and download the last versions of server and client for Windows. For me it was rabbitmq-server-2.7.1.exe and rabbitmq-dotnet-client-2.7.1-dotnet-3.0.zip. Client includes the WCF binding which I will use lately. There are several good manuals: rabbitmq-dotnet-client-2.7.1-user-guide.pdf and rabbitmq-dotnet-client-2.7.1-wcf-service-model.pdf. I was quite impressed by quality distributives and documentation.
  2. Start rabbitmq--server-x.x.x.exe. It will request to install Erlang. Agree, go to the Erlang site [http://www.erlang.org/] and download it. For me it was Erlang.otp_win32_R15B.exe
  3. Start Erlang…exe. Installation went smoothly.
  4. Start rabbitmq--server-x.x.x.exe again. Installation went smoothly.

RabbitMQ service is installed. You can see it in the Services window.

Creating RabbitMQ Client Base Assemblies

 Now it is time to install the client for .NET and WCF binding extension. Use this manual.

  1.  Downloaded the .msi. Installed it. Hmm… no Rabbit.Client.dll (When I started with Examples, they all need this file. It is a main client dll).

  2. Downloaded the *-dotnet-3.0.zip. Nope, it was wrong.
  3. We need the last rabbitmq-dotnet-client-2.7.1.zip file. Extract everything from this zip.
  4. Made a copy of the Local.props.example file to Local.props file in the same root folder.
  5. Start Visual Studio with RabbitMQDotNetClient.sln and successfully build it. The ..\rabbitmq-dotnet-client-2.7.1\projects\wcf\RabbitMQ.ServiceModel\obj\Debug\RabbitMQ.ServiceModel.dll was created. This dll together with the RabbitMQ.Client.dll are the assemblies used by the RabbitMQ clients.

Set up the RabbitMQ binding extension

Here I really lost a lot of time. It is easy for the C# clients; the RabbitMQ.ServiceModel.Test gives all the information. But we have to use this binding from the WCF-Custom LOB adapter. To do so we have to add the assemblies to GAC, change machine.config, etc. See the WCF LOP Adapter documentation.

  1. Sign the RabbitMQ.ServiceModel and RabbitMQ.Client projects and GAC-ed them.
  2. Insert  an <add> node into the config section of machine.config:

<extensions>
      <bindingExtensions>
        …
        <add  name="RabbitMQBinding" 
          type="RabbitMQ.ServiceModel.RabbitMQBindingSection, RabbitMQ.ServiceModel, Version=0.0.0.0, Culture=neutral,   PublicKeyToken=1751e286f1ab778d"/>
      </bindingExtensions>
    </extensions>

Note:

  • When the binding extension assembly is placed in GAC, we cannot use the above section in the app.config file! This <add> element should be placed only into the machine.config file. Otherwise it cannot be seen in the binding extension list of the WCF LOB Adapter.
  • Use >Gacutil.exe /l RabbitMQ.ServiceModel
    to get the real parameters for your assembly.
I have several machine.config files in folders:
    • C:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG
    • C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config
    • C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG
    • C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config

Which one to use? My VM is 64bit. BizTalk Host of the port is 32bit. BizTalk is the 2010 version. In my case the right file was the C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config.



Finally the RabbitMQ binding installed and we can see it in the Binding Type list for WCF-Custom adapter.

Testing

Creating the BizTalk test

I have created two pairs of port for testing.

File RP -> WCF-Custom (RabbitMQ) SP ->

 [RabbitMQ Server]

 -> WCF-Custom (RabbitMQ RP) -> File RP

 

I’ve changed only two parameters: “hostname” and “oneWay” and let others be default:

 

URL-s for the Send and Receive RabbitMQ Ports set up as “soap.amqp://192.168.11.128/Mytest” for both of them.

Let’s test it.

Small text file was dropped into In folder, consumed by Receive Port, went through the RabbitMQ queue, and appeared in Out folder. Success.

Note:

  • Now the RabbitMQ binding is set up to process only well-formed Xml documents.
  • “localhost” as “hostname” works fine.

Using the RabbitMQ Binding Element

The RabbitMQ binding is limited to the Text messaging encoding. If we need to change it or add more binding extensions, we have to use the customBinding type and use the rabbitMQBindingElement together with another element (including another encoding). How to do this?

1. Add the <add> node to the machine.config:

<bindingElementExtensions>
                           …
        <add name="rabbitMQBindingElement" type="RabbitMQ.ServiceModel.RabbitMQTransportElement, RabbitMQ.ServiceModel,   Version=0.0.0.0, Culture=neutral, PublicKeyToken=1751e286f1ab778d"/>
</bindingElementExtensions>

 2. Change the Receive Location transport parameters. It was the RabbitMQBinding. Change it to the customBinding. The Transport Binding Element has to be changed to the rabbitMQBindingElement. I changed only one parameter: hostname as “localhost”.

 

 3. Test it as previously.

Conclusion

Using RabbitMQ as one more transport for BizTalk and/or WCF applications is possible and straightforward. The learning curve is short. There are not too many issues with setting up the RabbitMQ Service and Client parts. Actually there wasn’t any RabbitMQ part which was exposed to the BizTalk directly. RabbitMQ works only with WCF-Custom adapter and only this adapter communicates with RabbitMQ Service.
Now the RabbitMQ implements limited client functionality for the WCF. RabbitMQ implements the AMQP v.0.9 protocol, not the last version 1.0.

Next steps

Next steps are to test this solution with different messaging patterns and try to figure out how to use RabbitMQ advantages (if there are some) over the MSMQ. Potentially the advantages are the simplest implementation of the Request/Response and Duplex patterns, better scalability, better performance, better manageability and support.

 

BizTalk and 0MQ (ZeroMQ)

It is stupid to compare those two beasts.

BizTalk Server is a pricey, huge development and run-time system, it includes tons of the tools, supports tons of protocols and systems. It works for very specialized middle ware integration stack. It integrates big and complex systems as SAP, CRM, Oracle and intermediates many protocols and standards as SOAP, EDI, SWIFT, FTP, tons of them.

It is a messaging system.

ZeroMQ is a low-level, wire-level protocol, the same level as TCP. It has ultra-simple API. It is implemented as a library for dozens languages as C#, Java, Python, PHP, C++, COBOL, Ada, Haskel, F#, Erlang, etc. It works in the most OS: Windows, Linux, QNX... It is free and works under open-source license.

It is a messaging system.

 

BizTalk implements many messaging exchange patterns: Request-Response, Pub-Sub, Queue, Message Broker, Router, Convoy.

0MQ implements the same: Request-Response, Pub-Sub, Pub-Sub with Topics, Queue, Message Broker, Router...

BizTalk development stack is compounded of several editors. BizTalk run-time stack includes several services on top of SQL Server, several tools, and tons of dll-s and SQL code.

0MQ is just a library, a dll. Library per language, per OS. No dedicated services.

...
Why I feel so comfortable with this 0MQ?
Why I think it is a serious choice to do messaging in my real-life messaging?
Do I want to abandon the comfort of a big track to ride almost bare-bone wheels?

I am not the only one who comparing 0MQ with big brokers. There is interesting article comparing RabittMQ (AMQP) and 0MQ.

BizTalk: Ordered Delivery

 

It is one more description of the Ordered Delivery (OD) in BizTalk.

The main article about it is in MSDN.

Here I am discussing the BizTalk Ordered Delivery “implementation details”.

OD Considerations

  • Ordered Delivery (sequential) mode is opposite of the “Parallel Delivery” mode. Parallel Delivery is the most productive mode; the Ordered Delivery is less productive mode.
  • Transports such MSMQ and protocols, supporting the WS-ReliableMessaging, are the protocols supporting OD. Other protocols as FTP, File, SQL, SMTP etc. do not have notion of the “order”.
  • BizTalk application usually is a part of the whole message workflow.
  • There are two approaches in the OD implementation:
    • All steps of the message workflow independently support OD.
    • A Destination System implements the re-sequencing and manages lost and duplicate messages.
  • Order is relative. Sequences can be ordered regards one or several parameters. For example, OD for the Company or for the Company + Department.

OD and the BizTalk Architecture

  • MessageBox is an implementation of the Message Queue. OD is an intrinsic feature of the MessageBox.
  • The BizTalk Server works in the Parallel Delivery mode by default.
  • There are three parts in the BizTalk message exchange outside of the MessageBox in relation to OD: Receive Locations; Orchestrations; Send ports.
    • Receive Locations support OD on the Transport level (say, MSMQ and some WCF adapters).
    • OD in Orchestrations is implemented by the sequential convoy pattern.
    • Send ports support OD for all static adapters.
  • The BizTalk Pipelines (part of Receive and Send Ports) always process messages in order using streams.

OD and Ports

To force Send Ports work in order we set up a flag the “Ordered Delivery” in Send Ports, then the BizTalk MessageBox takes care of implementing OD.

To force Receive Locations work in order we set up flag the “Ordered Delivery” option in the Receive Location Transports, whenever is possible. Then the BizTalk Transport Adapter takes care of implementing OD.

Ordered Delivery Send Port instance works as a singleton service. Since start it stays in Running state. It will not recycle if we restart its Host Instance. We could manually terminate it, if we want.

 

OD and Orchestrations

MessageBox implements the convoy message exchange pattern [See Using Correlations in Orchestrations]. See the detail convoy description in the BizTalk Server 2004 Convoy Deep Dive article.
It is not just a design pattern that developer should follow. There are special mechanisms inside MessageBox responsible for implementing OD.

OD and Orchestration: Sample

 

Imagine four Orchestrations which implement four approaches to the sequencing.



The first is the ProcessNoOrder Orchestration. It processes all messages without any order. One ProcessNoOrder Orchestration instance will be created for each inbound message.

The ProcessInOrder Orchestration processes all messages in one sequence. Only one ProcessInOrder Orchestration instance will be created.

The ProcessInOrderByCompany Orchestration processes messages in sequences correlated by the Company value (A, B, C, D, etc.). The separate queue is created for each new value of the Company. Messages inside queues are processed in order. Queues for different Companies are independent. A separate ProcessInOrderByCompany Orchestration instance will be created for each new Company value.

The ProcessInOrderByProduct Orchestration works exactly as the ProcessInOrderByCompany Orchestration but correlated by the Product value (xx, yy, etc.).

Discussions

Performance

By default all Orchestration and Messaging Service instances works in the Parallel Delivery mode with maximum performance.

If we check up the Ordered Delivery option in Send Port, BizTalk will initiate the Send Port instance as a singleton service. It is always a single instance. We don’t have the flexibility of the Orchestration where we could tune up “the proportion of the sequencing” and could control the recycling of the service instance.

Send Port OD could be in two states, on and off:

  • OD is off: a service instance per message, one message per queue, maximum performance.
  • OD is on: one infinite service instance, all messages in one queue, minimum performance.

Orchestration OD could be in two states also, on and off:

  • OD is off: a service instance per one activating message, one activating message per queue, maximum performance.
  • OD is on: one or several service instances, one per new correlation set value; all correlated messages per queue; performance is from min to max, depends ot the correlation set..

Carefully designing the correlation sets we could tune up the performance of the Orchestration. For example, if we only care of the document order for each separate Company, we include the Company to the Correlation set. If we had thousand documents related to hundreds companies, the performance will be near maximum. If there are only two companies, the performance will be near minimum, and we should consider improving the correlation with one more parameter.

Orchestrations and Zombies

Zombies are big problem of Convoy Orchestrations. See BizTalk: Instance Subscription and Convoys: Details article with description of this problem. This problem could be mitigated but could not be completely removed. We are waiting a new version of the BizTalk Server where this issue will be targeted.

BizTalk Server version Next, Ordered Delivery and Zombies

It is possible the BizTalk Server version Next will implement the automatic Ordered Delivery for Orchestrations, with pattern similar to the Ordered Delivery in Send Ports.

Three new Orchestration parameters are shown up there: Ordered Delivery, Stop on Exception, and Recycle Interval (in seconds).

 



Ordered Delivery parameter works in the same way as the Ordered Delivery parameter of the Send Port. Now we don’t have to build the Convoy Orchestration manually. No more Loop inside Orchestration.

If the Ordered Delivery parameter set up to True, the Orchestration is working as a Singleton. The first Receive shape receives all correlated messages in sequence. Correlation set is created implicitly regards of the Activation Subscription expression.

There are several limitations for this kind of Orchestration. The most important is: only one Receive shape is permitted here.

There are two big advantages of this new feature:

  • Simplified Orchestration design for the Ordered Delivery.
  • No more Zombies. The Orchestration instance is recycled in controllable way, when no messages, matched the Orchestration Subscription, are placed in the MessageBox.

Conclusion

We discussed the Ordered Delivery implemented in the BizTalk Server and ways to improve it.

 

BizTalk: Advanced Questions

This article is a Part 6 of the Interview Questions series.

It is time for fun!

These questions are for an Advanced level of the BizTalk Developers. I hope you don’t have a single chance to answer these questions, if you don’t have a real life experience in BizTalk Server projects.

If you have more advanced questions, please, comment them down and I will add them to this list under your name.

I will constantly add more questions into the end of list. Be in tune.
Recently I start adding new question only to the TechNet version of this article. So you also can add your questions there.

Advanced level

  • The error in BizTalk usually creates a suspended message. But routing error creates two messages. Why so? How we could utilize this feature?
  • Send ports have an “Ordered delivery” option. How it works for a File adapter, if the File protocol does not have a notion of file order?
  • Send ports have an “Ordered delivery” option but Receive ports do not have it. Why so?
  • You set up an “Ordered delivery” option in a Send Port. What is special about a Service Instance of this Port?
  • How to force an Orchestration do the ordered delivery? Is there a special “Ordered delivery” option in an Orchestration? If it is not, why?
  • Send ports have the “Priority” options, but Receive ports and Receive Locations do not have them. Why so?
  • How does the Priority feature of the Send Port work?
  • Why the Priority option can be set up only for the Send Port?
  • Send ports have the “Retry” options, but Receive ports and Receive Locations do not have them. Why so?
  • How does the Retry feature of the Send Port work? In what scenarios this feature is helpful, in
  • Send ports have a “Backup Transport” but Receive ports do not have it. Why so?
  • Send ports have a “Filter” but Receive ports do not have it. Why so?
  • Send ports have the “Maps” and “Transport”; Receive ports have “Maps” but “Transport” is associated with Receive Locations. Why so?
  • What is the difference between “Send Port Group – Send Port” relation and “Receive Port – Receive Location” relation?
  • Receive Locations have the Schedule parameters for date and time but Send Ports have the Schedule parameters only time. Why so?
  • Can we create two Send Ports with the same URL? Can we create two Receive Locations with the same URL? Why so?
  • Filter expression for the Send Port is defined at run-time, but Filter expression for the Orchestrations is defined only at design-time. Why so?
  • We created a custom pipeline component. Do we have to place it into the “<BizTalkFolder>\Pipeline Component” or install it to GAC or both?
  • How does an Orchestration make sure the message was sent successfully by one-way Send port? How does it changed for a two-way Send port?
  • What is it the “CodeList Database” property of the Schema? How to use it?
  • What is the “Property Schema Base” property of the Property schema element? It has the values: MessageDataProperyBase, MessageContextPropertyBase, and PartContextPropertyBase. What is the difference between using MessageDataProperyBase and MessageContextPropertyBase?
  • We bind an Orchestration with a Send Port. What does it do regards of the subscription?
  • We bind an Orchestration with a Receive Port. What does it do regards of the subscription?
  • How do Filters of the Send Ports and Send Port Groups work together? Say, a Send Port belongs to a Send Port Group. If this Send Port and this Send Port Group have the same Filter expression, does it create two subscriptions and, as a result, two messages are sent to the Send Port?
  • Why do we need a special Property schema for Promoted properties? What is the purpose of this schema? Distinguished properties do not use this kind of schema. Why so?
  • Properties of some BizTalk artifacts have a Tracking option. Right-click menu of some BizTalk artifacts have the “Tracking…” command. Why there is such a difference between artifacts? Why Maps do not have any Tracking options? Why Schemas have only Tracking property? Why Send Port Groups do not have any Tracking options? Why Receive Locations do not have any Tracking options? Why BizTalk Applications do not have any Tracking options?
  • When a BizTalk Assembly is deployed, is it stored in the BizTalk database or is it just referenced in the BizTalk database?
  • Why a BizTalk Assembly must be installed into GAC? Why deploying it to BizTalk database is not enough? 
  • Can you describe at least three scenarios where smart Orchestration Engine finds the errors in the messages patterns at design time (while build a BizTalk project)?
  • A Host has an "Authentication Trusted" option. What does it mean? What implication it has to the message processing? How do we have to use it / don't use?
  • Could we create an XLANGMessage in the C#/VB code? Please, provide code.
  • We can use an XmlDocument class as a Message type for receiving the untyped Xml messages in Orchestration. What .NET classes do we have to use for receiving the untyped text or binary messages in Orchestration. Why so?
  • Is the Publish-Subscribe BizTalk Engine API well defined? Where? If not, why so?
  • Does the BizTalk use the strategy “First  received, first processed” or “First received, last processed” of “First received, first processed, first sent” of something else? Why so?
    Describe scenarios, when this strategy works better, and scenarios, when this strategy works badly.
  • Are Orchestration Instances processed in order? Why so? Describe scenarios.
  • Is it possible to force the Hydration / Dehydration of the Orchestration instances, Messaging instances? How to do this? Why do this?
  • In what scenarios  will  we use the Compensations in Orchestrations? Why so?
  • The Atomic scopes in Orchestrations have a Retry feature. How does it work? Describe scenarios when it is used.
  • What serializer does the BizTalk use, an XmlSerializer or a DataContractSerializer? What implication has it for the real life scenarios?
  • What transform engine is the BizTalk using, an XslTransform or an XslCompiledTransform? What implication  has it for the real life scenarios?
  • The BizTalk Business Rule Engine uses a Rete engine. In what scenarios this Rete engine is the more efficient / less efficient?
  • Let’s classify the applications as Front-end, Middle-level and Back-end. Where are the best places to use BizTalk? Please, provide scenarios. What are the most important features we have to take in account?
  • What BizTalk features are the differentiators, when we compare the BizTalk Server with other alternatives? What are those alternatives?
  • What is a Heartbeat in the BizTalk? Why is it implemented? Can we change the heartbeat interval? How does this interval influence the processes in the MessageBox? 
  • What is the difference, if any, between the ESB and the Broker conceptions?
  • How to set up a custom name for a dynamic send port?
  • What kind of the dynamic ports are existed? Are there dynamic send  ports? Are there dynamic receive ports? Why is it so?

BizTalk: Samples: Error Handling, Email Notification

One problem with email notifications is it usually sends too many emails. Errors frequently happen in series. For example, a temporarily issue with network can result in hundreds repetitive emails. In result an email notification designed as a “VIP” notification is downgraded to the “annoying” status.

This email flooding could be mitigated with this solution.

A Sample demonstrates using the emails for sending notification in case of failed messages.

It compounds several similar errors in one email notification.

 

Recently I've uploaded this sample 

BizTalk: Samples: ErrorHandling: Notification emails

to MSDN Gallery.

It has all solution artifacts and can be used in real-life projects.