CloudCasts Blog

Webcasts in the Cloud
posts - 131 , comments - 71 , trackbacks - 120

My Links

News

Tag Cloud

Article Categories

Archives

Post Categories

Image Galleries

Bloggers Guides

Deadlettering in Azure AppFabric Service Bus Brokered Messaging

The following article will be included in the next release of “The Developer’s Guide to AppFabric”, a free e-book on Azure AppFabric development. The latest edition of the e-book is available for download here.
In a messaging system deadlettering is an action that is taken on messages, typically when they cannot be processed by a messaging system or by a receiving application. The Enterprise Integration Patterns book describes two patterns that can be used for scenarios where messages cannot be processed.
Dead Letter Channel
The dead letter channel pattern is used when a messaging system determines when it cannot process a message. This could be due to the message expiring by exceeding its time to live threshold, or by some error in the message processing system.

 

The Enterprise Integration Patterns website provides a description of the Dead Letter Channel pattern here.
 
Invalid Message Channel
The invalid message channel pattern is used when a receiving application decides that a message cannot, or should not be processed. This could be due to the message being formatted incorrectly, or not complying with business rules.

 

The Enterprise Integration Patterns website provides a description of the Invalid Message Channel pattern here.
 
Whilst very similar the key different in the two patterns is that the dead letter channel is used by the messaging system and the invalid message channel by the receiving application.
 
Deadlettering in AppFabric
In AppFabric brokered messaging both the dead letter channel pattern and the invalid message channel pattern are implemented on queues and subscriptions using deadletter sub-queues.
Messages are placed on the deadletter sub-queue by the messaging system in the following scenarios.
·         When a message expires and deadlettering for expired messages is set to true in a queue or subscription.
·         When the max delivery count for a message is exceeded on a queue or subscription.
·         When a filter evaluation exception occurs in a subscription and deadlettering is enabled on filter evaluation exceptions.
Receiving applications can also explicitly deadletter messages that have been received using the peek-lock receive mode by calling the Deadletter method on the Brokered message class, or client class. Once placed on the deadletter sub-queue the messages can be received by creating a QueueClient or MessageReceiver using the path for the deadletter sub-queue.
Deadletter Sub-Queues
AppFabric brokered messaging uses the prefix of $DeadLetterQueue for the path of the deadletter queues on topics and subscriptions. The deadletter sub-queue for the OrderQueue queue in the illustration will have the following path:

 

 
OrderQueue/$DeadLetterQueue
 
 
The deadletter sub-queues for the two subscriptions on the OrderTopic are as follows:

 

 
OrderTopic/Subscriptions/UKSubscription/$DeadLetterQueue
OrderTopic/Subscriptions/USSubscription/$DeadLetterQueue
 
 
The QueueClient and SubscriptionClient classes provide a static FormatDeadLetterPath method for returning the paths for the deadletter sub-queues for queues and subscriptions. The following code will get the appropriate paths.

 

 
// Get the deadletter sub-queue path for the OrderQueue
string queueDeadletterPath = QueueClient.FormatDeadLetterPath("OrderQueue");
 
// Get the deadletter sub-queue path for the UKSubscription on the OrderTopic
string subDeadletterPath =
    SubscriptionClient.FormatDeadLetterPath("OrderTopic", "UKSubscription");
 
 
Deadlettering Messages in Queues and Subscriptions
In the previous section it was stated that there are three scenarios where messages will messages will be deadlettered by the messaging system.
·         When a message expires and deadlettering for expired messages is set to true in a queue or subscription.
·         When the max delivery count for a message is exceeded on a queue or subscription.
·         When a filter evaluation exception occurs in a subscription and deadlettering is enabled on filter evaluation exceptions.
This section will focus on the second of these three scenarios and discuss how the use of dead-lettering and delivery counts can be used to handle service errors and “poison messages” when using AppFabric queues and topics.
MaxDeliveryCount Property
Every message in an AppFabric queue or subscription maintains a count of how many times that message has been received. The DeliveryCount property of the message starts at zero, and then increments by one each time a receiving application receives the message.
When a message is received from a queue by an application using the peek-lock receive mode the receiving application can perform one of the following operations.
·         Complete – The message is marked as complete and removed from the queue.
·         Abandon – Message processing is abandoned and is immediately available on the queue again.
·         Deadletter – The message is moved to the deadletter queue.
·         Differ – The message remains on the queue in the deferred state and can be received using the message sequence ID.
·         No action – The message will remain on the queue and become available on the queue again after the time interval specified in the LockDuration property of the queue.
The two operations that will allow the message to be received again in the normal way are Abandon and No action. Abandon is typically called by a receiving application if it cannot process a message and message processing is to be retried at a later point in time. If a receiving application terminates without calling one of the above operations on a message, then no action is taken.
Handling “Poison Messages”
In message oriented systems a “poison message” is a message that contains data or information that cannot be processed by the receiving application.
Suppose the following code is used to receive and process messages from a queue.

 

 
while (true)
{
    BrokeredMessage msg = queueClient.Receive();
    if (msg != null)
    {
        try
        {
            // Process message            
            msg.Complete();
        }
        catch (Exception ex)
        {
            // Log error
            msg.Abandon();
        }
    }
}
 
 
If the message is processed successfully it will be marked as complete. If an exception occurs during message processing then the exception will be handled, an error will be logged, and the message will be abandoned and returned to the queue and will be processed again.
If the message processing exception was related to a service call, or database operation, and there was a connection failure, the receiving application will process the message again and may well succeed on the second try. If, however, the exception was caused by the content of the message, the application will receive the message again, attempt to process it, catch an exception and abandon it, receive the message again, attempt to process it, catch an exception and abandon it, receive the message again, attempt to process it, catch an exception and abandon it. If there is no mechanism to detect that this is a “poison message” the cycle will continue indefinitely.
Poison Message Processing Example
In this example a receiving application will receive order statistics message and call a web service to calculate the price per item for the order. The web service that calculates this uses the following method.

 

 
public decimal Divide(decimal x, decimal y)
{
    if (DateTime.UtcNow.Millisecond % 2 == 0)
    {
        throw new ApplicationException
            ("Random internal service error.");
    }
 
    return x / y;
}
 
 
There are two exceptions that can occur when calling the service, a random internal error, which will happen about 50% of the time, and a divide by zero exception, which will happen if the value zero is passed as parameter y.
The receiving application will use the following code in its message receive loop.

 

 
while (true)
{
    Console.ResetColor();
    BrokeredMessage msg = queueClient.Receive();
    if (msg != null)
    {
        try
        {
            OrderStat stat = msg.GetBody<OrderStat>();
           
            Console.WriteLine("Order stat: Region - {0} {1} item(s), total ${2}",
                stat.Region, stat.NrItems, stat.OrderTotal);
            Console.WriteLine("DeliveryCount: {0}", msg.DeliveryCount);
 
            CalculatorClient client = new CalculatorClient();
 
            // Call the calculator service.
            decimal avgPricePerItem = client.Divide (stat.OrderTotal, stat.NrItems);
 
            Console.ForegroundColor = ConsoleColor.Cyan;
            Console.WriteLine("Avg price per item: " + avgPricePerItem);
            Console.WriteLine();
 
            // Mark the message as complete.
            msg.Complete();
        }
        catch (Exception ex)
        {
            // Log the error
            Console.ForegroundColor = ConsoleColor.Red;
            Console.WriteLine("Error: {0}", ex.Message);
            Console.WriteLine();
        }
    }
}
 
 
Note that abandon is not called on the message when handling the exception.
Testing with Valid Order Stats
When the application is tested with some sample order statistics the results are as follows.
Even though the first US order experienced processing errors twice due to an internal service error the message was received and processed successfully on the third attempt. The delivery count for the message is incremented each time the message is received. As the message was not abandoned when an exception occurred, the message remains on the queue in the locked state until the lock duration for the queue expires, which is 60 seconds be default. The application could be modified to abandon the message in the exception handling code as follows.

 

 
catch (Exception ex)
{
    // Log the error
    Console.ForegroundColor = ConsoleColor.Red;
    Console.WriteLine("Error: {0}", ex.Message);
    Console.WriteLine();
    msg.Abandon();
}
 
 
 
This would make the message available on the queue again, and it would be received and processed immediately.
To Abandon or not to Abandon?
Suppose the simulated internal service error is due to a failure in a system that the receiving application is attempting to communicate with. If this is the case it may be better to wait a while before trying to process the message again. If Abandon is called on the message the application will attempt to process the message again, of no action is taken there will be a delay based on the lock duration of the queue. In some scenarios it may be better to take no action when message processing fails, and rely on the lock duration providing a suitable retry interval. Be aware that other messages can be received and processed when a message is in the locked state, so this would mean that messages would not be processed on order.
Testing with Invalid Order Stats
If the application is testes with order statistics that specify zero items, a divide by zero exception will be thrown by the service. When the message processing is retried the same exception will be thrown again and again. A message that contains data that causes an exception when processing, or that fails to comply with business or formatting rules is known as a poison message.
When the application is tested with such an order statistic the results are as follows.
Instead of the retry process repeating indefinitely it stops when the delivery count of the message reaches 10. Queues and subscriptions have a MaxDeliveryCount property, which determines the number of times a message can be received before it is placed on the deadletter sub-queue. The default setting is 10, the minimum allowed value is 1 and the maximum allowed value is int.MaxValue, (2,147,483,647). Setting the maximum value effectively turns off the dead lettering of poison messages.
The MaxDeliveryCount is used as a means to prevent poison messages from blocking the receiving or messages from a queue or subscription.
Deadlettering Messages in a Receiving Application
It is also possible for a receiving application to deadletter messages if they are formatted incorrectly or fail to comply with business rules.
The brokered message class contains the following methods for deadlettering messages.

 

 
public void DeadLetter()
 
public void DeadLetter(string deadLetterReason, string deadLetterErrorDescription)
 
 
The QueueClient, SubscriptionClient and MessageReceiver classes provide the following messages.

 

 
public void DeadLetter(Guid lockToken)
 
public void DeadLetter
    (Guid lockToken, string deadLetterReason, string deadLetterErrorDescription)
 
 
When values are passed for the deadLetterReason and deadLetterErrorDescription parameters they will be added to the message properties collection with the names of DeadLetterReason and DeadLetterErrorDescription respectively. This will help a deadletter processing application to determine the cause of the failure.
Suppose the receiving application can only process order statistics for the UK, United States and Sweden.

 

 
List<string> validRegions = new List<string>()
{
    "UK", "US", "SE"
};
 
 
When a message is received a check could be performed to verify that the order statistics are for a valid region, if this validation check fails the message can be deadlettered by the receiving application. Explicitly deadlettering a message in the receiving application will save the message from being processed reputedly until it is deadlettered by the queue.

 

 
OrderStat stat = msg.GetBody<OrderStat>();
 
if (!validRegions.Contains(stat.Region))
{
    string errorDescription = string.Format("Region {0} is not known.", stat.Region);
    Console.WriteLine(errorDescription);
    msg.DeadLetter("Unknown region", errorDescription);
    msg.Dispose();
}
 
 
It is also possible for the receiving application to check the DeliveryCount property of a message and decide weather it should be abandoned or deadlettered.

 

 
if (msg.DeliveryCount < 3)
{
    msg.Abandon();
    msg.Dispose();
}
else
{
    msg.DeadLetter();
    msg.Dispose();
}
 
 
 
Receiving and Processing Deadlettered Messages
Messages that are deadlettered will remain on the deadletter sub queue. Is queues and topics in AppFabric have a maximum size it is critical that and deadlettered messages are removed and processed. Failure to do so will result in queues and topics refusing to accept new messages when that size threshold is reached. The same techniques used for receiving messages form queues and subscriptions can be used for receiving messages from deadletter sub-queues.
When receiving messages from a deadletter sub-queue the DeliveryCount property of the message will not be incremented. When received using the default peek-lock receive mode messages can be completed, abandoned or deferred, but not deadlettered. Attempting to call the Deadletter method on a message received from the deadletter queue will result in an exception.
The following code shows a simple message receive loop that will receive messages from the OrderStatQueue deadletter sub-queue and display diagnostic information including the reason and error message for deadlettering the message.

 

 
string deadLetterQueuePath = QueueClient.FormatDeadLetterPath("OrderStatQueue");
 
QueueClient deadletterQueueClient = factory.CreateQueueClient(deadLetterQueuePath);
 
while (true)
{
    BrokeredMessage msg = deadletterQueueClient.Receive();
 
    if (msg != null)
    {
        Console.WriteLine("Deadlettered message.");
 
        Console.WriteLine("MessageId:                  {0}", msg.MessageId);
        Console.WriteLine("DeliveryCount:              {0}", msg.DeliveryCount);
        Console.WriteLine("EnqueuedTimeUtc:            {0}", msg.EnqueuedTimeUtc);
        Console.WriteLine("Size:                       {0} bytes", msg.Size);
        Console.WriteLine("DeadLetterReason:           {0}",
            msg.Properties["DeadLetterReason"]);
        Console.WriteLine("DeadLetterErrorDescription: {0}",
            msg.Properties["DeadLetterErrorDescription"]);
        Console.WriteLine();
        msg.Complete();           
    }
}
 
 
When the application is tested with a poison message, and a message with an invalid region the output from the deadletter processing console is as follows:
 
Receiving Deadlettered Messages from Subscriptions
Whilst using a queue client combined with the static FormatDeadLetterPath method to receive messages from the deadletter sub-queue of a queue, the same technique will not work with subscriptions. If a topic named OrderTopic contains a subscription named UKSubscription the following code can be used to get the path for the deadletter sub-queue of the subscription.

 

 
SubscriptionClient.FormatDeadLetterPath("OrderTopic", "UKSubscription");
 
 
The following value will be returned.

 

 
OrderTopic/Subscriptions/UKSubscription/$DeadLetterQueue
 
 
Creating a SubscriptionClient to receive messages from this sub-queue can be done in two ways.

 

 
// Create a Message Receiver
string deadletterPath =
    SubscriptionClient.FormatDeadLetterPath("OrderTopic", "UKSubscription");
MessageReceiver deadletterReceiver = Factory.CreateMessageReceiver(deadletterPath);
 
// Create a SubscriptionClient
SubscriptionClient deadletterSubscriptionClient = Factory.CreateSubscriptionClient
    ("OrderTopic", "UKSubscription/$DeadLetterQueue");
 
 
 
It is possible to use the value returned by SubscriptionClient.FormatDeadLetterPath to create a MessageReceiver, but not a SubscriptionClient.
Monitoring Deadlettered Messages
The SizeInBytes and MessageCount properties of queues, topics and subscriptions will include any deadlettered messages that are present on the deadletter sub-queues of queues and subscriptions. There is, however, no way to tell how many deadlettered messages there are, or the size consumed by these messages.

Print | posted on Friday, October 21, 2011 11:22 AM |

Feedback

No comments posted yet.
Post A Comment
Title:
Name:
Email:
Comment:
Verification:
 
 

Powered by: