BizTalk: Compensation Model
There are still many questions in using Compensation in BizTalk.
· What is the proper order of the Exception and Compensating blocks (handlers), Compensate shapes?
· Do we have to rethrow the Exceptions?
· Do we have to use Compensate shape for the current transaction or for the internal transactions?
· What exactly does the Compensate shape?
· What is the difference between Atomic and Long-Running transactions from the Compensation point of view?
Let’s start from the theory.
Compensation and Rollback
I have to use term the Rollback in this document. It is not from the BizTalk Documentation.
Data_1 --> [Transaction: Data_1to2] --> Data_2
I use the Rollback term like the same term in the database transactions. Rollback means restoring all data to the point before transaction. It is completely automatic, atomic process. We cannot control rollback. Only one operation is possible, Start Rollback. It rolls back only the local orchestration data, orchestration variables and parameters.
The local orchestration data outside the transaction could be only Data_1 or Data_2. When Transaction is Committed, the data is Data_2. When Transaction is Rollbacked, the data automatically returns to the pre-transaction state, to the Data_1. We never get the Data_1to2 outside the Transaction. We all know this; it is ABC of the Transaction model.
But BizTalk is not a data base server as SQL Server and transaction in BizTalk is not a data base transaction. For example the Send shape inside transaction scope can send message to the external system and change data in this system. In general case BizTalk cannot automatically roll back this operation and won’t do this. We have to manually create a code to roll back the external changes. Atomic transaction might change the external system in the same way as Long-Running transaction. BizTalk helps us with managing this roll back procedure. We include the custom roll back code in the Compensation block. This code is invoked automatically by BizTalk or manually by a Compensate shape and the whole process called the Compensation.
Compensation it is not a Rollback. It is another process.
Compensation is a controlled process. Here is a typical example: code inside BizTalk transaction inserts rows in the database. BizTalk does not automatically rollback this operation. But it gives us possibility to implement [maybe] complex business process of deleting inserted rows. Plus, the last uncontrolled step of the Compensation is a Rollback. We implement some custom business process of compensation and then BizTalk invoke Rollback.
All Rollbacks and Compensations are initiated by an Exception. If an Exception is fired inside transaction, BizTalk invokes Rollback, but Compensation for this transaction is usually prohibited.
BizTalk invokes Rollback, but before that BizTalk invokes the Exception block of this transaction, if there is such block and if this block catches the Exception.
Note: Compensation block and Compensation shape are different things. Compensation shape is a method to call the Compensation block.
Atomic and Long-Running transactions
By design Atomic transaction should be used for the fast and simple procedures. The data before Atomic transaction is not persisted on the disk. Atomic transaction is quick that’s why it does not damage the total reliability. You can mention that Atomic transaction property the Timeout automatically set up to 60 sec, but not for the Long-Running transaction. Moreover only the Atomic transaction has the Property the Retry because it makes sense to automatically retry only the short process not the long one.
The likelihood of the system crash is bigger in the durations of hours and days than in seconds. That’s why data before Long-Running transaction persisted to disk. That’s why the orchestration data used in the Long- Running transaction must be serializable. That’s why the Long-Running transaction theoretically slower than Atomic transaction.
Note: Atomic transaction in BizTalk is not atomic in strict sense. It can include requests to the external processes, calls to the .NET classes, etc. Each of these processes could change something outside the orchestration and the transaction Rollback does not automatically roll back all these changes. It would be dangerous.
The Atomic transaction is short by definition. It cannot nest other transactions inside. The Atomic transaction is processed as a single unit of work. If Exception is fired inside the Atomic transaction, all we have to do is to roll back this transaction. And BizTalk makes this automatically. We cannot change this roll back process no matter what sort of Exception was fired.
Seems this is the main difference between n the Atomic and Long-Running transactions in BizTalk, how they behave in case of the internal Exception. Atomic transaction always automatically performs Rollback. We cannot avoid this Rollback and cannot do something else. Long-Running transaction could catch the Exception in the optional Exception block and process the rollback in specific custom code. But the Exception block cannot prevent the automatic Rollback. The Rollback occurs after executing the Exception block. If a Long-Running transaction does not have the Exception blocks it processes an Exception the same way as an Atomic transaction.
Compensations and Exceptions
Rule 1: Transaction block can get only one custom Compensation block.
Rule 2: If the Compensation block is not defined, the Transaction uses the default Compensation. Default Compensation is the Compensation for the current Transaction. That means the unwiring the Compensations for the nested Transactions and Rollback (restoring the Orchestration variables).
Rule 3: Firing an Exception is the only way to initialize the Compensation process.
Rule 4: Atomic transaction block cannot get the Exception block.
Rule 5: Long-Running transaction use the default Exception block for the “General Exception” if the custom Exception block for the “General Exception” is not defined. Default Exception block invokes the Compensation shape for current transaction and rethrow the Exception.
Rule 6: Compensation shape can be inserted only inside Exception block or Compensation blocks.
It seems obvious but anyway…
We can include the Compensate shape nowhere but only in the Exception and Compensation blocks. We cannot start the compensation process arbitrary. No exception – no compensation! Sometimes we need to utilize the automatic compensation ability of the BizTalk in the ordinary business process. Firing Exception is not the ordinary situation; an exception should be always indicator of the error in process. But here we have to use exception, there are no other ways.
Rule 7: Transaction that fired the Exception processes a Rollback right after this Exception. We cannot compensate this transaction.*
*) As Charles Young mentioned here is a workaround for this rule (see the “Using Compensation Blocks on L-R Transactions” topic in his article), but you have to use undocumented features of the BizTalk. BizTalk is wisely trying to prevent compensating uncommitted transaction.
Standard compensation process (Example)
Let’s cover the standard compensation process.
Below is the simple orchestration with one LR nesting transaction (1---) and two nested transactions (11… and 12…).
Second nested transaction fired the Exception.
Let see what is going on here:
1. Code generates the DivideByZeroException
2. The Exception is caught in the Exception block of the same Transaction.
3. The Exception is rethrown.
4. BizTalk rolls back the Transaction, that means it restore the orchestration variables and parameters.
5. The Exception is caught in the Exception block of the nesting, outer Transaction.
6. Calls to the Compensating of the nested Transactions in the revers order. Compensation of the 12… cannot be executed because this transaction was not Committed (completed).
7. Compensating of the 11… Transaction.
8. Exception is rethrown. Orchestration is suspended.
Why the Compensation blocks the Comp12 and Comp 1 were not executed? For different reasons.
Comp12 was not executed because Transaction the 12… was not committed but rollbacked and a try to call the Compensation in the C_Ex 1 block was skiped.
Comp1 was not executed because it could be called only from the outer, nesting transaction. We don’t have such transaction here.
The trace log was:
1 ---
1 --- 11...
1 --- 11... 12... (1)
1 --- 11... 12... >>> Exc 12 (2)
1 --- 11... 12... <<< Exc 12 Rethrow Exception ******** (3) (4)
1 --- 11... >>> C_Ex 1 (5)
1 --- 11... >>> Comp 11 (6)
1 --- 11... <<< Comp 11 (7)
1 --- 11... <<< C_Ex 1 Rethrow Exception ******** (8)
Below is the XLANGs code generated by BizTalk Orchestration Editor:
module GLD.Samples.Compensation
{
internal service longrunning transaction StandardCompensation
{
port implements PortType_1 Control_R;
message System.Xml.XmlDocument msg_XmlDocument;
System.Int32 var_int;
System.String var_State;
body ()
{
activate receive (Control_R.Operation_1, msg_XmlDocument);
var_int = 0;
var_State = "";
scope longrunning transaction tx_1
{
System.Int32 var_IntLevel1;
body
{
var_IntLevel1 = 0;
var_State = "1 --- ";
System.Diagnostics.Trace.WriteLine(var_State);
scope longrunning transaction tx_11
{
body
{
var_State = var_State + "11... ";
System.Diagnostics.Trace.WriteLine(var_State);
}
compensation ()
{
System.Diagnostics.Trace.WriteLine(var_State + " >>> Comp 11");
System.Diagnostics.Trace.WriteLine(var_State + " <<< Comp 11");
}
}
scope longrunning transaction tx_12
{
body
{
var_State = var_State + "12... ";
System.Diagnostics.Trace.WriteLine(var_State);
// Exception!!!
var_int = var_int / System.Convert.ToInt32("0");
}
exceptions
{
catch
{
System.Diagnostics.Trace.WriteLine(var_State + " >>> C_Ex 12");
System.Diagnostics.Trace.WriteLine(var_State + " <<< C_Ex 12 Rethrow Exception ********");
throw;
}
}
compensation ()
{
System.Diagnostics.Trace.WriteLine(var_State + " >>> Comp 12");
System.Diagnostics.Trace.WriteLine(var_State + " <<< Comp 12");
}
}
}
exceptions
{
catch
{
System.Diagnostics.Trace.WriteLine(var_State + " >>> C_Ex 1");
if (succeeded(tx_12))
{
compensate tx_12 ();
}
if (succeeded(tx_11))
{
compensate tx_11 ();
}
System.Diagnostics.Trace.WriteLine(var_State + " C_Ex 1 Rethrow Exception ******** ");
throw;
}
}
compensation ()
{
System.Diagnostics.Trace.WriteLine(var_State + " >>> Comp 1");
compensate tx_1 ();
System.Diagnostics.Trace.WriteLine(var_State + " <<< Comp 1");
}
}
}
}
The uncear points in the Compensation Model
To me there are several confusing things in the Compensation model.
First, the Compensation blocks are called outside. Event to call the Compensation comes outside! It is pretty unusual from the developer experience. One more time: the Compensation block is called outside. We are waiting that methods start from the first operator, and the exceptions catch blocks are placed in the end of code. The Exception blocks in the Orchestration are placed as we are waiting for, in the end of the code. Compensation blocks placed in the same place but worked in opposite manner. It is unintuitive. Now the similar graphic presentation of the Exception and Compensation blocks mixed with different behaviour of the Exception and Compensation blocks. It is misleading.
Second, Rollbacks and Compensation are kind of unsynchronized. I will tell about this in the next sample.
Compensation Model process (Example)
Let’s cover the more complex compensation process.
This is the transaction tree:
Below is the orchestration picture. Nesting transaction was implemented in nested loops.
Exception was fired in the 1222 transaction.
The trace log was:
------------------------------- 5 ----------------------------
1000***
1000*** 1100===
1000*** 1100=== 1110---
1000*** 1100=== 1110--- 1111...
1000*** 1100=== 1110--- 1111... 1112...
1000*** 1100=== 1110--- 1111... 1112... 1120---
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121...
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122...
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200===
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210---
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211...
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212...
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... 1220---
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... 1220--- 1221...
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... 1220--- 1221... 1222...
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... 1220--- 1221... 1222... ******* Throw exception(1)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... 1220--- 1221... >>> C_Ex 3(2)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... 1220--- 1221... >>> Comp 4(3)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... 1220--- 1221... <<< Comp 4(4)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... 1220--- 1221... <<< C_Ex 3 ******* ReThrow exception(5)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... >>> C_Ex 2(6)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... >>> Comp 3(7)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... >>>Comp 4(8)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... <<< Comp 4(9)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... >>> Comp 4(10)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... <<< Comp 4 (11)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... <<<Comp 3(12)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... 1200=== 1210--- 1211... 1212... <<< C_Ex 2 ******* ReThrow exception(13)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... >>> C_Ex 1(14)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... >>> Comp 2(15)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... >>> Comp 3(16)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... >>> Comp 4(17)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... <<< Comp 4(18)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... >>> Comp 4(19)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... <<< Comp 4(20)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... <<< Comp 3(21)
1000*** 1100=== 1110--- 1111... 1112... >>> Comp 3(22)
1000*** 1100=== 1110--- 1111... 1112... >>> Comp 4(23)
1000*** 1100=== 1110--- 1111... 1112... <<< Comp 4(24)
1000*** 1100=== 1110--- 1111... >>> Comp 4(25)
1000*** 1100=== 1110--- 1111... <<< Comp 4(26)
1000*** 1100=== 1110--- 1111... 1112... <<< Comp 3(27)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... <<< Comp 2(28)
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122... <<< C_Ex 1(29)
.......................................................................
1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122...
Picture of whole compensation process:
As an initial exception (1) is fired, there are several stages in the compensation process. A catch block follows after each exception, then compensations (blue arrow) and rollbacks (green arrow) for nested transaction(s). Each compensation is followed by a rollback with nested compensations and rollbacks in between.
For example, an Exception (5) in the Catch Exception block the “C_Ex 3” of the transaction 1220 is caught into the Catch Exception block (6) of the Transaction 1200, then follows the Compensation “Comp3” for the Transaction 1210 (7), the Compensation (8) and the Rollback (9) for the Transaction 1212, the Compensation (10) and the Rollback (11) for the Transaction 1211, the Rollback (12) for the Transaction 1210.
You can mention one interesting detail in the whole compensation process: All compensation blocks were called and all compensation code inside was invoked. But the orchestration data was not returned to the initial state. The orchestration data was roll backed only for the transaction branch where exceptions were fired, for the transaction 1200 and nested transactions. We could predict the roll backed data would be the “” but it is “1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122...”
Is it a bug in BizTalk?
Seems the compensation process is tuned up only for the “default” compensation model for some reason. In this model each implicit Exception block is finished by a Throw Exception operator. As you can remember the Default Exception block invokes the Compensation shape with current transaction and rethrow an initial Exception. If we follow this model we have got a predictable rollback process.
In the preceding example we have to nest the Transaction 1000 in outer Transaction and add a Throw Exception operator in “C-Ex 1” Exception block. In this case the data would be roll backed to the “” value in this outer transaction.
Rule 7: The orchestration data is roll backed to the initial point of the transaction where the last exception is rethrown.
For example, the last exception was rethrown in the Transaction 1200. The initial data for this Transaction was “1000*** 1100=== 1110--- 1111... 1112... 1120--- 1121... 1122...”. The orchestration data was roll backed to this value, not to the initial data for the outer compensated Transaction 1000.
Conclusion
Let’s summarize how to create the standard compensation process.
1. Define the Compensation blocks for all transaction you want to be compensated. This is the most chalenging task.
2. Define the Exception blocks where the Exception(s) should be cought.
3. Add the Compensate shape(s) to this Exception block. Define what transaction you want to compensate. The standard way is to define a current transaction. All nested transactions would be compensated automaticaly in the right order.
4. If you need a predictable rollback process for the orchestration data, rethrow an Exception in each Exception block where you added the Compensate shape.