Introduction
Perhaps the greatest architectural change between BizTalk Server 2004 and earlier versions is the introduction of the MessageBox and the subscription mechanism that delivers messages to instances of BizTalk services. Subscription is central to understanding BizTalk Server 2004, and is the subject of this article. In many ways, it is a fairly straightforward and simple mechanism, but there are subtleties. This article explains the fundamental features of subscription, and explores its implementation within the BizTalk engine. In a future article, I will cover additional features of the subscription mechanism such as ordered delivery, convoying, message predicates etc.
The essential idea of subscription is very simple. Messages are received (via receive handlers or from orchestration Send ports) and handed to the BizTalk engine. The engine determines and stores a set of contextual property values for the message, and then interrogates a rule store to determine a set of matching subscriptions. Each subscription comprises a set of rules which match messages in terms of property values. For each matched subscription, a record is entered into an application-specific queue, and is associated with a specific instance of a service. Queued messages are then de-queued using a number of separate worker threads, and routed to the designated instances of services. In essence, the subscription mechanism acts as a rules engine that infers, from a set of predicate-based rules, which messages should be handed to which service instances.
Fig. 1: BizTalk Server 2004 Subscription
Before we go further, I should mention that BizTalk Server 2004 ships with a rules engine codenamed 'Dragonfly'. When I began to look at the mechanisms involved in subscription, I discovered that the Subscription Viewer (a tool in the SDK) uses a specialised class (SubscriptionRuleStore) provided by this rules engine to represent subscription rule-sets programmatically. I got quite excited about this discovery, assuming that BizTalk subscription is implemented using the Dragonfly rules engine. However, after further research, I can find no evidence that this is the case. The subscription mechanism is largely implemented using a number of SQL Server stored procedures to match subscriptions to messages and to queue these messages. The subscriptions, themselves, have a number of BizTalk-specific features, and the SubscriptionRuleStore class does not fully represent these features. Hence the Subscription Viewer does not provide a comprehensive insight into subscriptions, although the main features are covered.
Message Properties
The subscription mechanism primarily matches subscriptions to messages in terms of contextual message properties. Each time a message is received by BizTalk, its properties are determined, and are recorded in the MessageProps table. As we will see the second article, not all message properties are promoted into the MessageProps table in this way. For those that are, some of these property values are provided automatically by BizTalk and others are derived from fields within the message itself.
The MessageProps table serves something of a double purpose. As well as holding the promoted contextual properties of each message, it is also used to record the batch of messages each individual message belongs to, and the order in which the message was received within the batch. In BizTalk Server 2004, all received messages belong to a batch. However, some adapters only support batches containing exactly one message.
Subscription Matching
Once BizTalk has recorded the properties of an inbound message, it next finds all subscriptions which match the incoming message using the bts_FindSubscriptions stored procedure. Each subscription is defined within the Subscription table as a separate record with a subscription ID (a Guid) and a creation time. Subscriptions may be grouped together using a Group ID, and each subscription group has a priority number (see the section on priority below).
Predicates
Subscriptions are defined primarily in terms of message property predicates. Each predicate describes an expression used in subscription matching. For example, a predicate might define an expression that tests for messages with an orderQty property value that is greater than 5000. BizTalk supports the following predicate types:
- Bitwise AND
- Equals
- Exists
- GreaterThanOrEquals
- GreaterThan
- LessThanOrEquals
- LessThan
- NotEquals
The Subscription Viewer utility does not represent Bitwise AND or Exists predicates directly. Instead, these are represented as Equals predicates. In the case of Exists predicates, the Subscription Viewer represents these as ' = EXISTS'. 'Exists' predicates test for the existence of a message property rather than its value. A good example of the use of 'Exists' predicates is on subscriptions created by dynamic send ports. When a dynamic send port is enlisted, it creates separate subscriptions for each of the adapters registered with the system. Each subscription includes an 'Exists' predicate to check for the existence of an 'OutboundTransportLocation' message property.
Predicates are collected into ‘And’ and ‘Or’ groups. BizTalk does not allow these groups to be nested, and therefore is somewhat constrained in terms of the subscription rules which potentially could be created.
Services, Service Instances and Ports
Subscriptions are created by services, and results in messages being routed to service instances. It is important to explain carefully the concept of services in BizTalk. As we will see, this is not quite as straightforward as you might expect. The term 'service' has more than one meaning.
BizTalk runs as a Windows service (BTSNTSvc.exe). A Windows service is, of course, a software program that runs in its own process outside of the current logged-on user's security context. One of the tasks of this service is to manage BizTalk hosting. A BizTalk host is a configurable and administrable logical unit within BizTalk that, at runtime, maps to one or more 'host instances'. A host instance manages loaded BizTalk components such as adapters. A BizTalk host virtualises host instances, allowing single point administration of a scalable group of host instances running on a cluster (group) of BizTalk machines. BizTalk defines two types of host. ‘In-process’ hosts are used to create host instances that run within a BTSNTSvc.exe process. ‘Isolated’ hosts define host instances that run within other processes, such as IIS-defined processes.
Host instances are used to host various types of BizTalk component including 'Host Instance Sub-services'. These sub-services are COM components that implement the undocumented IBTSService interface through which sub-service instances can be started and stopped. There are five BizTalk sub-services defined as follows:
- Caching Service
- End Point Manager
- MSMQT
- Tracking
- XLANG/s
The caching and tracking sub-services are treated as internal to BizTalk, whereas other sub-services provide the messaging and orchestration facilities directly available to BizTalk developers and administrators. The End Point Manager (EPM) sub-service supports general messaging between external endpoints. The MSMQT sub-service supports the use of the message queuing adapter to provide an MSMQ native interface integrated directly with the BizTalk MessageBox. Both EPM and MSMQT allow adapters and pipelines to be configured and enlisted via the notion of Receive Ports/Receive Locations and Send Ports. The XLANG/s sub-service manages business process execution using orchestrations.
Fig. 2: BizTalk Server 2004 Hosts and Sub-Services
For the purposes of subscription, tracking and activity monitoring, BizTalk provides a further definition of the term 'service'. For the sake of clarity, I will call these 'subscriber services' in order to distinguish them from Windows services or host instance sub-services. However, they are simply called 'services' within the BizTalk documentation. Subscriber services are closely related to host instance sub-services. Host instance sub-services provide the functionality that defines subscriber services and which create and manage instances of those subscriber services at runtime.
A subscriber service is modelled along the same general lines as a service described by WSDL (Web Service Description Language). It acts as a collection of one or more ports. Each port is an endpoint to which messages are routed or from which messages are returned to the MessageBox. BizTalk categorises subscriber services in terms of 'service classes'. There are four service classes defined as follows in the adm_ServiceClass table in the administration database:
- Messaging InProcess
- Messaging Isolated
- MSMQT
- XLANG/s
Note that the two Messaging service classes are supported by the EPM host instance sub-service. HAT provides a slightly different view of services classes, and recognises some additional 'internal' classes. These include the 'Tracking' service class, and a catch-all 'Other' class. HAT also recognises a 'Routing Failure Report' service class. When the subscription mechanism cannot successfully route a message to a port on an instance of a subscriber service, BizTalk generates a routing failure report message which is posted to the MessageBox. This message is not subscribed to by any actual service. The 'Operations/Messages' menu in HAT can be used view messages currently in the MessageBox. This view displays all messages and their service classes, including the Routing Failure Report messages.
At run time, the host instance sub-services create and manage instances of the subscriber service classes. These 'service instances' then receive messages from the MessageBox via the subscription mechanism (and de-queuing), and may return messages to the MessageBox. The notion of service classes and instances is somewhat abstract, but relates directly to runtime service classes and instances. This can most clearly be seen in terms of XLANG/s orchestrations. Orchestrations are defined in a BizTalk Visual Studio project, and are compiled into .NET classes that implement the BTXService interface (this is not related to the IBTSService interface mentioned above). At run time, the XLANG/s sub-service creates instances of these classes. Each instance is an in-memory state machine that receives, processes and returns messages to and from the MessageBox. Messages are delivered by the subscription mechanism directly to these service instances. In a similar fashion, each time a Send port is created through the BizTalk Explorer, a new EPM service is defined. Run-time instances will then be created to handle messages routed to the Send port.
At run time, any single service class may have several instances. Each instance is recorded, for the duration of its lifetime, within the Instances table. In the case of Orchestrations, of course, this lifetime may be extended by dehydrating the instance. Each instance can be in one of several states as follows:
- Ready to Run
- Active
- Dehydrated
- Suspended (Not Resumable)
- In Breakpoint
- Zombie.
Service instances and ports are the key to understanding subscriptions. The BizTalk engine uses each subscription to route messages to ports on specific service instances. Subscriptions may, or may not, contain an Instance ID. Subscriptions with an Instance ID are related directly to a live or dehydrated service instance. Orchestration service instances create these instance-specific subscriptions dynamically. Each time process flow reaches a Receive shape in an orchestration, subscriptions are dynamically created for the orchestration ports connected to that shape. The subscription is removed when a message has been received on the port. Subscriptions without an Instance ID result in the creation of a new service instance each time they match an incoming message, and are termed 'Activate' subscriptions.
Each subscription may also be associated with a port. Ports are defined at the service level. In the case of EMP 'Messaging' services, there is one port per service. In the case of Orchestration services, each orchestration can define multiple ports. Orchestrations allow one port to be marked as the 'Activate' port' This results in the publishing of a single 'Activate' subscription which will result in new orchestration instances being created each time the subscription matches an inbound message. Note that subscriptions are created for EPM 'Send' ports and orchestration 'Receive' ports. These are treated in the same way by the subscription mechanism. In earlier versions of BizTalk, we got used to thinking of messaging ports and orchestration ports as very different things, but in BizTalk Server 2004, although they are implemented by different code, they are unified in terms of subscription.
As well as services of the four service types described at the beginning of this section, BizTalk creates subscriptions for the additional 'internal' services. For example, the Cache Manager creates a single subscription that selects messages with a 'BizTalkControl' property with a value of 'CacheRefresh'. BizTalk uses the MessageBox to route internal control messages.
Subscription Evaluation and Message En-queuing
Each time a matching subscription is found for an inbound message, the BizTalk engine evaluates that subscription. Evaluation is a simple process that, after some checks, inserts the message into an application-specific queue. In the second article, we will explore the way in which the evaluation stage checks for the existence of message predicates.
In BizTalk, each application is defined by a separate 'Host', and by set of queue tables within the MessageBox. As well as the main queue table, these include the 'suspended' and 'scheduled' queues. As we have seen, a 'host' is a virtual container that maps to one or more 'host instances' at runtime. Only the Enterprise version of BizTalk supports multiple host instances for any one host, and each host instance runs on a separate server within a BizTalk group. For the purposes of our discussion, we are chiefly interested in the fact that messages are placed on the main queue in a specific order reflecting the order in which messages were received, and that each subscription defines a priority level which is used to set the priority of the message within the queue.
Note that the queue contains only various identifiers for the message and the service instance. Additional data, including the message context, is spooled into a separate table (Spool) with individual message parts stored in the MessageParts table.
Message De-queuing
BizTalk reads messages off the queue using one or more 'de-queuing' worker threads. One of these threads is designated as the ‘primary’ thread. The de-queuing threads deliver (route) each message to the designated service instance. As you can imagine, there is quite a lot of fine detail involved in this process. BizTalk uses locks to ensure that there is no contention between the threads, and that deadlocks cannot occur. De-queuing is completely de-coupled from message delivery. The BizTalk engine constantly inspects queues to discover and process any new messages. Each time BizTalk invokes a de-queuing stored procedure, it inspects the queue for messages that belong to a particular service class, and which are destined for a specific host instance (for subscriptions which are instance-specific), and reads off a batch of messages. The maximum size of a batch is 20 records.
Priority
Message batches are de-queued in priority order. Priority is defined at the subscription level, and ranges from 1 to 10. 1 represents the highest priority and 10 represents the lowest. The default priority level is 5. BizTalk allows you to set the priority level when configuring EPM messaging Send ports, but not when configuring Orchestration ports. Orchestration ports generate subscriptions with a priority of 7. Because messages are de-queued in service class-specific batches, priority levels do not operate across different service types. It is not relevant, for example, if an Orchestration port has a lower priority that an EPM Messaging port. However, if one EPM Messaging port has a higher priority than another, this is highly relevant. Messages will be delivered to service instances in priority order.
Unfortunately, the SDK subscription viewer does not display the priority data for subscriptions, even though it has this information available to it through the SubscriptionRuleStore class.
Since we are discussing priority, it is worth mentioning another aspect which currently I do not fully understand. Towards the beginning of this article I mentioned briefly that subscriptions are grouped, and that each subscription group has a group priority. When BizTalk matches subscriptions to messages using the bts_FindSubscriptions stored procedure, it primarily orders the subscriptions by the order in which the matching messages are received in the message batch, and then secondarily by the subscription group ID (a Guid). The records are then further ordered by subscription group priority. If we assume that subscriptions are then evaluated in this sort order, this will affect the order in which messages are placed on the queue. However, it is difficult to see what ordering by group priority achieves. Unfortunately I have yet to see a case where there are multiple subscriptions in a single subscription group, so I cannot tell for certain how this feature is used.
Conclusions
This article has discussed the basic mechanism of subscription. There are a number of additional features which will be the subject of an additional article. These include ordered delivery, convoying, filtering, message predicates and time windows.