Geeks With Blogs

News
Charles Young

Most developers understand the classical 'ACID' concept of a transaction as an atomic unit of work that, when performed, moves the system from one consistent state to a new, consistent and durable state, and which is isolated from other transactions.  In BizTalk orchestrations, this kind of transaction is called 'atomic'.   It is associated, in a one-to-one fashion with Scope shapes, and with the Catch Exception and Compensation blocks attached to the scope    Scope shapes could more accurately be called 'Context' shapes.   They define a context which maintains state for a defined unit of orchestration code, and this can include transaction state.

 

Long-Running Transactions

 

BizTalk also supports another, non-ACID, type of transaction.   Long-running transactions are also defined units of work, but unlike atomic transactions, they cannot be isolated from other transactions.   Like atomic transactions, they are associated on a one-to-one basis with Scope shapes.  As all BizTalk developers know, long-running transactions are vitally important in business process orchestration, where a unit of work may take seconds, hours or even days to complete.  Microsoft's classic explanation of L-R transactions used to be that they are just like ACID transactions, but without 'I' (Isolation).   I note that in the BizTalk 2004 documentation, they now say that L-R transactions also lack 'A' (atomicity), but maintain 'C' (consistency) and 'D' (durability).   In a general sense, I am very uncomfortable with this highly simplistic type of explanation. The truth is that ACID properties are all deeply inter-connected.   To affect one is to affect them all.  How could it be claimed, for example, that L-R transactions maintain consistency in the strong ACID sense in an environment where transactions are not isolated from each other and may take arbitrary times to complete, or may not complete at all?  Consistency, however, can be achieved in L-R transactions through the mechanism of compensation.  However, this offers a significantly 'weaker' (no flame mail, please - you all know what I mean) approach to maintaining consistency than using, say, serialized atomic transactions.

 

Microsoft's claim that BizTalk L-R transactions maintain consistency and durability does, however, accurately express characteristics which arises from the way orchestration code interacts with the BizTalk engine.  Each Scope shape in the orchestration maps to a discrete segment of code.  The chief characteristics of these segments are that they mark the boundaries where control passes back to the BizTalk engine which persists data to the MessageBox, leaving it is a consistent and durable state.   BizTalk L-R transactions do indeed exhibit consistency and durability in terms of MessageBox state.  However, the code enclosed within an L-R transaction may interact with all kinds of external transactional systems.  It is here that consistency and durability may be compromised, which is why atomic transactions scopes can be nested within L-R transactions.

 

Atomic Transactions

 

Atomic transactions cannot contain nested transactions or Catch Exception blocks.  They also cannot contain Send and Receive shapes bound to the same two-way port.   Orchestration ports are non-transactional (perhaps this will change in future versions), and therefore could compromise the atomicity of transactions.   Atomic transactions do support three additional features that are not supported by L-R transactions

 

  • The 'Isolation Level' property defines the degree of isolation between the state changes performed by different atomic transactions.   L-R transactions do not support transaction isolation, and therefore do not support this property.   BizTalk supports three isolation levels.  These are 'Read Committed', 'Repeatable Read' and 'Serializable'.   Earlier versions of BizTalk also offered 'Read Uncommitted' (inherited from the COM+ model), but this has been dropped in BizTalk 2004.

 

  • The 'Retry' property, which is true by default, specifies that the atomic transaction can be retried in the event of a failure.  To perform a retry, throw an instance of Microsoft.XLANGs.BaseTypes.RetryTransactionException within the transaction boundary.   This exception class has a DelayFor property which you can set to a TimeSpan to control the backoff interval.  Be careful of this feature.   You can easily create a transaction which continuously fails and retries, forming a type of never-ending loop (the actual behaviour of the BizTalk engine is to retry continuously for 20 seconds and then suspend the Orchestration).  Some readers may again notice that this approach is markedly different to earlier versions of BizTalk which supported the declarative configuration of retry counts and backoff intervals.   In BTS2K4 you program this in yourself.

    There is a single remark in the help files that says there are occasions when the Orchestration engine itself will attempt to retry atomic transactions.   This opaque remark is unqualified, and I certainly cannot find any place where backoff intervals or retry counts can be set at the engine level.   I cannot tell you anything further about this issue, but you should be aware that retries may possibly be attempted automatically.

 

  • The 'Batch' property, which is true by default, "determines if this transaction can be batched with other transactions across multiple instances of the orchestration" according to Microsoft.   Unfortunately, the exact behaviour of batched atomic Orchestration transactions is not documented further in the woefully incomplete help files.   I have far more questions than answers in this area, so the following section represents my current speculations.  Any suggestion I make may be entirely inaccurate. 

 

Batching Transactions

 

Let’s start by seeing what Microsoft has to say about the Batch property.   Here is the sum total of their documentation on the subject:

 

"Batch - Determines whether an orchestration that is an atomic transaction can be batched with other instances."

 

"Batch - Boolean value that determines if this transaction can be batched with other transactions across multiple instances of the orchestration. The default value is True. "

 

"Batch property - From the drop-down list, select whether an atomic transaction scope can be batched with other atomic transactions across multiple instances of the orchestration."

 

"Batch - Indicates if this atomic transaction should be processed in a batch.  May improve performance with the possibility of losing isolation data."

 

There's not much to work on here. The last of the four quotations is not even strictly from the documentation.  It is the description shown in property box in the IDE.   The first quote, incidentally, is from a help page that specifically describes Orchestration properties, rather than the properties of a Scope shape.   The entire Orchestration is wrapped in a scope that can be transactional.

 

I shall make the assumption, which appears to be supported by the second and third quotations above, that batching of atomic transactions applies only to transactions in multiple instances of the same Orchestration.   This batching may, I suspect, be related to batched message submission.  A Receive adapter always submits messages to the BizTalk engine as part of a batch.  In most cases, adapters create batches consisting of just one message, but an adapter can potentially submit multiple messages as a single batch.   Another feature of a Receive adapter is that it can hand off DTC (Distributed Transaction Co-ordinator) transactions to the engine.   Hence, if anything goes wrong while processing the batch within BizTalk, the engine can (I assume) vote on the outcome and the DTC transaction can be rolled back.   The BizTalk messaging engine additionally notifies the adapter about the outcome of the entire batch. The following annotated quotation is highly relevant:

 

"After the [Receive adapter] batch has been processed, the transport proxy [an object that acts as a proxy of the adapter transport to the BizTalk engine] invokes the adapter callback method, the BatchComplete method of the IBTBatchCallBack interface [i.e., calls back into the Receive adapter]. The status of the submission is passed to the adapter as an array of HRESULTS corresponding to each message in the batch. If the batch fails, either in the pipeline or in the orchestration [my emphasis], the SOAP fault message is returned to the adapter as a response."

 

The logical conclusion to draw is that the concept of batched processing extends through BizTalk all the way to Orchestrations.  If a batch consists of multiple messages, it is logical to assume that multiple instances of an Orchestration may be invoked to process these messages due to subscription mechanism (unless, of course, correlation is used to convoy the messages through a single Orchestration instance - convoying is a particularly powerful technique which I may write about at a later date).  It is further logical to assume, given the documentation, that if any one of these Orchestration instances fails, a DTC transaction representing that entire batch can be rolled back, and the adapter is notified.   In other words, if a single Orchestration instance fails, the entire batch fails.

 

It is an obvious step to speculate further that there may be a tie-up between the notion of batched atomic Orchestration transactions and the DTC transactions provided by the a Receive adapter with a batch of messages.  Perhaps the notion of 'batching' of transactions across multiple Orchestration instances only applies across those instances which are processing messages received in a single batch.  It might be that the Batch property may explicitly allow atomic transactions within single instances of an orchestration to vote on the outcome of any DTC transaction provided by a Receive adapter - perhaps if this property is set to 'false', the DTC transaction may complete, even if an atomic Orchestration transaction fails.   This would be a useful level of fine-grained control in batch processing scenarios.

 

Of course, I cannot tell for sure from the sparse documentation.  In order to find out, I will need to build a test adapter that submits batched messages wrapped in a DTC transaction, and then experiment with the Batch property to see what happens.  This failure by Microsoft to document the behaviour of BizTalk adequately is a really annoying time-waster.

 

Another area open to speculation is what it actually means for one "transaction [to] be batched with other transactions across multiple instances of the orchestration".   Does this mean that there is some additional batch wrapper built around multiple separate transactions (for what purpose?) or does it mean that, in effect, the corresponding transactional blocks in multiple Orchestration instances are enlisted into a single transaction?   I cannot tell, but I have a suspicion that this second approach is meant.  This would neatly support the statement that batched transactions "may improve performance with the possibility of losing isolation data.   It could improve performance by reducing the overall number of transactions at run-time.   However, it would bind apparently different transactions in different Orchestration instances into one single transaction, therefore compromising the expected isolation of the activity in each separate instance.

 

There is further evidence that that this may be exactly what is meant by 'batching transactions'.   In my research into this issue, I noticed that there is an Orchestration engine performance counter called 'Average Batch Factor'.  This is described as follows:

 

"Number of persistence points reached since the host instance started, divided by the number of underlying transactions."

 

This counter would make perfect sense in a situation where multiple atomic transactions are merged into a single transaction by the Orchestration engine.   It would (assuming that a 'persistence point' is really a segment boundary) provide some hard facts about the effect of batching atomic transactions.  The greater the number, the greater the effect of 'transaction batching' (i.e., the less underlying transactions are being created).

 

If I am right in my speculation, we should be aware of the issue.  The potential for loss of data consistency in external systems, coupled with the potential for deadlock and race conditions, could possibly be very significant in batched message scenarios.  By failing to document clearly and adequately the transaction behaviour of BizTalk, Microsoft may be condemning some BizTalk developers to huge amounts of additional stress, endless late evenings and long weekends while they try to discern and fix strange, unexpected behaviour within their Orchestration code.   Of course, I may be entirely wrong, in which case I would like to know just exactly what the Batch property is really all about.   Please tell us, Microsoft.  We really need to know.

 

DTC-based Transactions

 

Earlier versions of BizTalk 2002 exploited COM+ extensively and directly to support both atomic (DTC) transactions and L-R transactions.  L-R transactions were nothing more than an application of the COM+ Compensating Resource Manager (CRM) feature.   Indeed, the best documentation on L-R transactions was contained in the COM+ documentation!  Indeed, the entire XLANG engine was simply a COM+ component.

 

This is no longer the case within BizTalk Server 2004.   Microsoft states in its documentation that Orchestration transactions are not, by default, DTC transactions (they would be, if they were COM+ (Enterprise Services) based). 

 

Despite this, BizTalk Server 2004 does understand COM+ DTC transactions.   According to the documentation, "you can explicitly make [atomic transactions into] DTC transactions, provided that any objects being used in the transaction are COM+ objects derived from System.EnterpriseServices.ServicedComponents, and that isolation levels agree between transaction components.   I can find no additional mechanism for specifying the use of DTC, so my assumption is that if you use Enterprise Services objects of the same isolation level within an atomic transaction, that transaction automatically supports DTC.   The Timeout property of atomic transaction Scope shapes directly maps to COM+, and has no effect if the transaction is not participating in a DTC transaction.

 

Another thing to be aware of is that the DTC is used to manage distributed transactions within a BizTalk Server group.  If DTC is not enabled on group servers, the Configuration Framework will fail.  This is well documented by Microsoft.

 

 

Posted on Sunday, April 25, 2004 8:42 PM | Back to top


Comments on this post: Transactions in BizTalk Server 2004 Orchestrations

# Transaction in BizTalk Server 2004 Orchestrations
Requesting Gravatar...
BizTalk 2004 supports long-running and atomic transactions within Orchestrations. The model has changed a bit from earlier versions, and atomic transactions are no longer based on DTC transactions by default. This article explains some of the features of transactions, and also speculates about the exact behaviour of 'batched' atomic transactions - a subject that is currently very opaque due to a lack of documentation.
Left by Charles Young on Apr 25, 2004 3:48 PM

# BizTalk 2004 Transactions
Requesting Gravatar...
Definitely a good read. Raises quite a few interesting questions!...
Left by Commonality on Apr 26, 2004 8:17 PM

# re: Transactions in BizTalk Server 2004 Orchestrations
Requesting Gravatar...
I might say that Long Running transactions are NOT ACID transactions but CD only (consistency and durablity). You mention that only the Isolation is missing, in my opinion the Atomicity is also missing.
Left by Nick Makris on Feb 25, 2005 8:33 AM

# re: Transactions in BizTalk Server 2004 Orchestrations
Requesting Gravatar...
oopps forget it, you have it just a few lines below.
Left by Nick Makris on Feb 25, 2005 8:34 AM

# re: Transactions in BizTalk Server 2004 Orchestrations
Requesting Gravatar...
As for the Atomic shape, we now know that the BTS will retry till 20 attempts in case of failure and then the orchestration will enter the Suspended (resumable) state. However, does this also apply to the loop shape as well ?
I have a loop shape inside an atomic scope and for no good reason, the orchestrations always enter the Suspended (resumable) state. Does this occur because it loops for too long ?
Any ides ?
Left by Sanket Bakshi on May 16, 2005 8:12 AM

# re: Transactions in BizTalk Server 2004 Orchestrations
Requesting Gravatar...
Regarding my previous query with the Atomic Shape, we figured out that this is a bug with BTS 2004. MS has provided us with a post SP1 patch. The problem was that the atomic scope was not able to write its state to the database. Hence, by rule, after 20 retries, the orchestration went into a suspended (resumable) state.
Left by Sanket Bakshi on May 24, 2005 4:49 PM

# re: Transactions in BizTalk Server 2004 Orchestrations
Requesting Gravatar...
HI Charles,

You are very knowledgeable in long running transactions in BizTalk. But I have bigs problems understanding it. Could you please kindly clarify the following points of mine.

1. When a long running transaction can't commit, all of the compensation blocks of its nested transaction will be called automatically. right? There will be No automatice rollback with the current transaction. Right?
2. in Biztalk documentation, there is a line "If you choose to compensate the current transaction, that means that the default handler will be invoked, and not an explicit compensation block (if there is one)." This got me really confused. Is there really a way you can choose to compensate the current transaction? How?
3. In BizTalk documentation, "you can add a Compensate shape in the compensation block or an exception block of a transaction scope. " So, if compensation blocks of the nested transactions will be called automaticlly if an exception occurs (if I undersatnd it correctly), I can't think of any situation where a compensate shape needs to added in an exception block.

Your help is greatly appreciated!
teddy
Left by teddy on Nov 08, 2005 9:04 PM

# re: Transactions in BizTalk Server 2004 Orchestrations
Requesting Gravatar...
Guyz,

In Long transaction at BizTalk 2004, have any of you met some conditions that you need to modify the orchestration? perhaps because the business requirements changes and we need to add / remove some steps at the orchestrations. But when we need to redeploy the modified orchestration, the instances has to be removed first at the biztalk.
Is there any way besides the side by side execution for this issue?
Has biztalk 2006 already resolved this issue?
Thanks all
Left by Bembeng Arifin on Dec 07, 2005 8:33 AM

# re: Transactions in BizTalk Server 2004 Orchestrations
Requesting Gravatar...
Not really. Because BizTalk artefacts are generated as C# source behind the scenes) and compiled into .NET assemblies, the only way to manage deployment of new code in the presence of existing long-running transactions is through side-by-side assembly deployment. You have to ensure that the old versions of the assemblies are there to service the existing transactions when they are re-hydrated. After all, a dehydrated orchestration is simply a seep serialisation of an object graph, including the objects in your BizTalk artefacts, custom code, etc. This does not change in BTS 2006

There are possible 'alternative' strategies. For exampel, you might be able to break your orchestration functionality into finer-grained, discrete 'steps', and adopt a step management design pattern where some central orchestration is used to control next steps. We have done this using the Rules Engine to determine next steps. The Rules Engine uses the notion of business Policies to control the publication of versioned rule sets, and you can have multiple rules set versions published side-by-side. However, this approach also opens up the possibility of being able to change the flow of existing long-running business activities dynamically by replacing one version of a policy with another, and this could allow you some flexibility for retiring .NET code 'early'. There are lots of potential issues, though, and huge scope for ending up with a mess. I wouldn't recommend going down this route.
Left by Charles Young on Dec 07, 2005 9:24 AM

Your comment:
 (will show your gravatar)


Copyright © Charles Young | Powered by: GeeksWithBlogs.net