Most developers understand the classical 'ACID' concept of a transaction as an atomic unit of work that, when performed, moves the system from one consistent state to a new, consistent and durable state, and which is isolated from other transactions. In BizTalk orchestrations, this kind of transaction is called 'atomic'. It is associated, in a one-to-one fashion with Scope shapes, and with the Catch Exception and Compensation blocks attached to the scope Scope shapes could more accurately be called 'Context' shapes. They define a context which maintains state for a defined unit of orchestration code, and this can include transaction state.
Long-Running Transactions
BizTalk also supports another, non-ACID, type of transaction. Long-running transactions are also defined units of work, but unlike atomic transactions, they cannot be isolated from other transactions. Like atomic transactions, they are associated on a one-to-one basis with Scope shapes. As all BizTalk developers know, long-running transactions are vitally important in business process orchestration, where a unit of work may take seconds, hours or even days to complete. Microsoft's classic explanation of L-R transactions used to be that they are just like ACID transactions, but without 'I' (Isolation). I note that in the BizTalk 2004 documentation, they now say that L-R transactions also lack 'A' (atomicity), but maintain 'C' (consistency) and 'D' (durability). In a general sense, I am very uncomfortable with this highly simplistic type of explanation. The truth is that ACID properties are all deeply inter-connected. To affect one is to affect them all. How could it be claimed, for example, that L-R transactions maintain consistency in the strong ACID sense in an environment where transactions are not isolated from each other and may take arbitrary times to complete, or may not complete at all? Consistency, however, can be achieved in L-R transactions through the mechanism of compensation. However, this offers a significantly 'weaker' (no flame mail, please - you all know what I mean) approach to maintaining consistency than using, say, serialized atomic transactions.
Microsoft's claim that BizTalk L-R transactions maintain consistency and durability does, however, accurately express characteristics which arises from the way orchestration code interacts with the BizTalk engine. Each Scope shape in the orchestration maps to a discrete segment of code. The chief characteristics of these segments are that they mark the boundaries where control passes back to the BizTalk engine which persists data to the MessageBox, leaving it is a consistent and durable state. BizTalk L-R transactions do indeed exhibit consistency and durability in terms of MessageBox state. However, the code enclosed within an L-R transaction may interact with all kinds of external transactional systems. It is here that consistency and durability may be compromised, which is why atomic transactions scopes can be nested within L-R transactions.
Atomic Transactions
Atomic transactions cannot contain nested transactions or Catch Exception blocks. They also cannot contain Send and Receive shapes bound to the same two-way port. Orchestration ports are non-transactional (perhaps this will change in future versions), and therefore could compromise the atomicity of transactions. Atomic transactions do support three additional features that are not supported by L-R transactions
- The 'Isolation Level' property defines the degree of isolation between the state changes performed by different atomic transactions. L-R transactions do not support transaction isolation, and therefore do not support this property. BizTalk supports three isolation levels. These are 'Read Committed', 'Repeatable Read' and 'Serializable'. Earlier versions of BizTalk also offered 'Read Uncommitted' (inherited from the COM+ model), but this has been dropped in BizTalk 2004.
- The 'Retry' property, which is true by default, specifies that the atomic transaction can be retried in the event of a failure. To perform a retry, throw an instance of Microsoft.XLANGs.BaseTypes.RetryTransactionException within the transaction boundary. This exception class has a DelayFor property which you can set to a TimeSpan to control the backoff interval. Be careful of this feature. You can easily create a transaction which continuously fails and retries, forming a type of never-ending loop (the actual behaviour of the BizTalk engine is to retry continuously for 20 seconds and then suspend the Orchestration). Some readers may again notice that this approach is markedly different to earlier versions of BizTalk which supported the declarative configuration of retry counts and backoff intervals. In BTS2K4 you program this in yourself.
There is a single remark in the help files that says there are occasions when the Orchestration engine itself will attempt to retry atomic transactions. This opaque remark is unqualified, and I certainly cannot find any place where backoff intervals or retry counts can be set at the engine level. I cannot tell you anything further about this issue, but you should be aware that retries may possibly be attempted automatically.
- The 'Batch' property, which is true by default, "determines if this transaction can be batched with other transactions across multiple instances of the orchestration" according to Microsoft. Unfortunately, the exact behaviour of batched atomic Orchestration transactions is not documented further in the woefully incomplete help files. I have far more questions than answers in this area, so the following section represents my current speculations. Any suggestion I make may be entirely inaccurate.
Batching Transactions
Let’s start by seeing what Microsoft has to say about the Batch property. Here is the sum total of their documentation on the subject:
"Batch - Determines whether an orchestration that is an atomic transaction can be batched with other instances."
"Batch - Boolean value that determines if this transaction can be batched with other transactions across multiple instances of the orchestration. The default value is True. "
"Batch property - From the drop-down list, select whether an atomic transaction scope can be batched with other atomic transactions across multiple instances of the orchestration."
"Batch - Indicates if this atomic transaction should be processed in a batch. May improve performance with the possibility of losing isolation data."
There's not much to work on here. The last of the four quotations is not even strictly from the documentation. It is the description shown in property box in the IDE. The first quote, incidentally, is from a help page that specifically describes Orchestration properties, rather than the properties of a Scope shape. The entire Orchestration is wrapped in a scope that can be transactional.
I shall make the assumption, which appears to be supported by the second and third quotations above, that batching of atomic transactions applies only to transactions in multiple instances of the same Orchestration. This batching may, I suspect, be related to batched message submission. A Receive adapter always submits messages to the BizTalk engine as part of a batch. In most cases, adapters create batches consisting of just one message, but an adapter can potentially submit multiple messages as a single batch. Another feature of a Receive adapter is that it can hand off DTC (Distributed Transaction Co-ordinator) transactions to the engine. Hence, if anything goes wrong while processing the batch within BizTalk, the engine can (I assume) vote on the outcome and the DTC transaction can be rolled back. The BizTalk messaging engine additionally notifies the adapter about the outcome of the entire batch. The following annotated quotation is highly relevant:
"After the [Receive adapter] batch has been processed, the transport proxy [an object that acts as a proxy of the adapter transport to the BizTalk engine] invokes the adapter callback method, the BatchComplete method of the IBTBatchCallBack interface [i.e., calls back into the Receive adapter]. The status of the submission is passed to the adapter as an array of HRESULTS corresponding to each message in the batch. If the batch fails, either in the pipeline or in the orchestration [my emphasis], the SOAP fault message is returned to the adapter as a response."
The logical conclusion to draw is that the concept of batched processing extends through BizTalk all the way to Orchestrations. If a batch consists of multiple messages, it is logical to assume that multiple instances of an Orchestration may be invoked to process these messages due to subscription mechanism (unless, of course, correlation is used to convoy the messages through a single Orchestration instance - convoying is a particularly powerful technique which I may write about at a later date). It is further logical to assume, given the documentation, that if any one of these Orchestration instances fails, a DTC transaction representing that entire batch can be rolled back, and the adapter is notified. In other words, if a single Orchestration instance fails, the entire batch fails.
It is an obvious step to speculate further that there may be a tie-up between the notion of batched atomic Orchestration transactions and the DTC transactions provided by the a Receive adapter with a batch of messages. Perhaps the notion of 'batching' of transactions across multiple Orchestration instances only applies across those instances which are processing messages received in a single batch. It might be that the Batch property may explicitly allow atomic transactions within single instances of an orchestration to vote on the outcome of any DTC transaction provided by a Receive adapter - perhaps if this property is set to 'false', the DTC transaction may complete, even if an atomic Orchestration transaction fails. This would be a useful level of fine-grained control in batch processing scenarios.
Of course, I cannot tell for sure from the sparse documentation. In order to find out, I will need to build a test adapter that submits batched messages wrapped in a DTC transaction, and then experiment with the Batch property to see what happens. This failure by Microsoft to document the behaviour of BizTalk adequately is a really annoying time-waster.
Another area open to speculation is what it actually means for one "transaction [to] be batched with other transactions across multiple instances of the orchestration". Does this mean that there is some additional batch wrapper built around multiple separate transactions (for what purpose?) or does it mean that, in effect, the corresponding transactional blocks in multiple Orchestration instances are enlisted into a single transaction? I cannot tell, but I have a suspicion that this second approach is meant. This would neatly support the statement that batched transactions "may improve performance with the possibility of losing isolation data.” It could improve performance by reducing the overall number of transactions at run-time. However, it would bind apparently different transactions in different Orchestration instances into one single transaction, therefore compromising the expected isolation of the activity in each separate instance.
There is further evidence that that this may be exactly what is meant by 'batching transactions'. In my research into this issue, I noticed that there is an Orchestration engine performance counter called 'Average Batch Factor'. This is described as follows:
"Number of persistence points reached since the host instance started, divided by the number of underlying transactions."
This counter would make perfect sense in a situation where multiple atomic transactions are merged into a single transaction by the Orchestration engine. It would (assuming that a 'persistence point' is really a segment boundary) provide some hard facts about the effect of batching atomic transactions. The greater the number, the greater the effect of 'transaction batching' (i.e., the less underlying transactions are being created).
If I am right in my speculation, we should be aware of the issue. The potential for loss of data consistency in external systems, coupled with the potential for deadlock and race conditions, could possibly be very significant in batched message scenarios. By failing to document clearly and adequately the transaction behaviour of BizTalk, Microsoft may be condemning some BizTalk developers to huge amounts of additional stress, endless late evenings and long weekends while they try to discern and fix strange, unexpected behaviour within their Orchestration code. Of course, I may be entirely wrong, in which case I would like to know just exactly what the Batch property is really all about. Please tell us, Microsoft. We really need to know.
DTC-based Transactions
Earlier versions of BizTalk 2002 exploited COM+ extensively and directly to support both atomic (DTC) transactions and L-R transactions. L-R transactions were nothing more than an application of the COM+ Compensating Resource Manager (CRM) feature. Indeed, the best documentation on L-R transactions was contained in the COM+ documentation! Indeed, the entire XLANG engine was simply a COM+ component.
This is no longer the case within BizTalk Server 2004. Microsoft states in its documentation that Orchestration transactions are not, by default, DTC transactions (they would be, if they were COM+ (Enterprise Services) based).
Despite this, BizTalk Server 2004 does understand COM+ DTC transactions. According to the documentation, "you can explicitly make [atomic transactions into] DTC transactions, provided that any objects being used in the transaction are COM+ objects derived from System.EnterpriseServices.ServicedComponents, and that isolation levels agree between transaction components.” I can find no additional mechanism for specifying the use of DTC, so my assumption is that if you use Enterprise Services objects of the same isolation level within an atomic transaction, that transaction automatically supports DTC. The Timeout property of atomic transaction Scope shapes directly maps to COM+, and has no effect if the transaction is not participating in a DTC transaction.
Another thing to be aware of is that the DTC is used to manage distributed transactions within a BizTalk Server group. If DTC is not enabled on group servers, the Configuration Framework will fail. This is well documented by Microsoft.