nServiceBus

Distributing WCF service load vertically as well as horizontally

Elton Stoneman — Fri, 16 Dec 2011 07:52:12 GMT

Overview

I've been looking into a solution recently which has a series of remote nodes publishing events to a central node. The publishers have no logic, they just need to contact the central node and record that an event has happened. Off the back of that, the central node does some work inferring what the event means, and computing its relationship to a bunch of other events. Simple enough, but the solution needs to be scalable and make efficient use of its resources.

I started off building it around NServiceBus, but when I had a better understanding of the components and the physical architecture, decided on a different approach. The central node will most likely be an EC2 cluster; using NServiceBus, cluster nodes can communicate with each other's queues, but the remote nodes would need a bridging mechanism to talk to the central node, and I'd rather have a consistent communication model. So I started rebuilding the service layer in WCF, but I wanted flexibility on how the central cluster nodes distributed work amongst themselves.

Solution

Briefly, I wanted a nice solution for this scenario: remote nodes will always talk to the central cluster over HTTP with a REST service; central nodes will distribute load amongst themselves as efficiently as they can. In a low-load situation, the node receiving the originating message should do all the work in the workflow. In a high-load situation, the node receiving the originating message should farm out work to the other nodes. EC2 will let you set up a nice load balancer, but the option to balance load between nodes OR within a node needs a bit more thought - i.e. when a cluster node receives an event published message, it can call the next service in the workflow either by remotely sending a WCF service request to the load balanced URL, or by executing the code locally.

This is actually very simple, and I've published a working example on my github code gallery: DistributedServiceSample. In the sample, all service calls in the central node are run through a generic service invoker:

Service .Execute< IComputeService >(svc => svc.Compute(jobId));

The invoker either builds a WCF client proxy, or instantiates the service implementation locally, and makes the call. In the sample the logic for deciding whether to go remote or local is done in config, but this could be worked into something more complex based on current capacity etc. The sample also allows the invoker to decide whether to make a synchronous call, or farm the work out to an async task (again, through config).

Dependencies

Dynamically building a WCF client proxy is very simple, there are no dependencies outside .NET and you can leave all the endpoint configuration to the normal <serviceModel> config section. When the central node is in remote mode, it gets the proxy like this:

var factory = newChannelFactory<TService>("*");
service = factory.CreateChannel();

(The asterisk tells WCF to pull the client endpoint and binding from config based on the contract name and assumes there is only one client entry per contract).

When in local mode, it's a little bit more involved to do it dynamically. In the sample I have a marker interface (IService) to denote a service contract. In the service application startup I register all service implementations in an IoC container (a wrapper around Unity), and then the central node gets the service like this:

service =

Container.Get<TService>();

The actual method call on the service is done through functions or actions (depending on whether the service returns a response), so it's all typesafe.

Running the sample

Build the sample locally, and open http://localhost/DistributedServiceSample.Services/JobService.svc in WCFStorm or soapUI. Call CreateJob with whatever parameter you like, and you will see output similar to this in DebugView:

JobService.CreateJob called with Name: fhwfiy
Service.GetExecutionMode using *Synchronous* for service: DistributedServiceSample.Contracts.Services.IJobService
Service.GetServiceLocation using *Remote* for service: DistributedServiceSample.Contracts.Services.IJobService
JobService.SaveJob called with jobId: fhwfiy
Service.GetExecutionMode using *AsynchronousIgnoreResponse* for service: DistributedServiceSample.Contracts.Services.IComputeService
Service.GetServiceLocation using *Local* for service: DistributedServiceSample.Contracts.Services.IComputeService
ComputeService.Compute called with jobId: 621849676

The workflow is that CreateJob triggers a SaveJob call, which in turn triggers a Compute call. The service decides whether each downstream call will be made locally or remotely, synchronously or asynchronously based on the contents of the <distributedservicesample.invoker> section in Web.config.

Extensions

The obvious extension is to add an operation name to the config settings, so different operations within the same service can be executed in different ways, which is pretty straightforward. More complex is the idea of dynamically deciding whether to make a local or a remote call. The logic for the decision is all isolated, so it would be a case of swapping out the config stuff with some environment checks, so calls were made locally unless CPU or private bytes or current connections were above a threshold.

Processing Excel uploads with BizTalk and nServiceBus

EltonStoneman — Sat, 21 Nov 2009 01:57:56 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

Following on from the sample for processing Excel uploads with nServiceBus, I have a comparable version using BizTalk on MSDN Code Gallery here: BizTalk and nServiceBus Excel Upload. The BizTalk (2006 R2) sample uses a FILE receive port with a simple pipeline component to disassemble the Excel file into separate messages. A SQL send port subscribes to the messages, and with an outbound map calls the AdventureWorks stored procedure to insert each product.

Processing Excel in BizTalk is nothing new, but I wanted to do a comparison against the nServiceBus example, and also see how BizTalk and nServiceBus could be integrated. My original thinking was that the trigger in the nServiceBus solution relies on a FileSystemWatcher, which is less reliable and less flexible than BizTalk’s FILE adapter. A hybrid solution could use the FILE adapter to receive and parse the upload, then send each row as an AddProduct message to MSMQ, which the nServiceBus handler subscribes to.

Running this on the same environment (a Windows Server 2003 VM running under VirtualBox in Ubuntu 9.04), the BizTalk solution processes the 3,500 row Excel file in 3 mins 10 seconds (compared to 4m 15s for the distributed NSB running with 5 threads), and the 12,000 row file in 11m 14s (compared to 14m 0s). I was surprised to find the BizTalk solution running more quickly, as the original NSB was using non-recoverable messaging, so all messages were in memory, while BizTalk had the latency of writing to the message box.

Performance Comparison

This is not an intended to be a thorough benchmark of NSB and BizTalk – both sample projects are basic, untuned implementations, and the tests are on a single box rather than two or three. But for comparison, I ran the 3,500 row upload repeatedly under different configurations:

1. NSB with distributor – host and distributor using 20 threads

2. NSB with distributor – host and distributor using 20 threads, recoverable messaging

3. NSB without distributor – host using 20 threads

4. NSB without distributor – host using 20 threads, recoverable messaging

5. BizTalk FILE receive and SQL send

6. NSB parsing Excel file, BizTalk subscribing to AddProduct messages – MSMQ receive using batches of 100

7. BizTalk parsing Excel file, NSB without distributor subscribing to AddProduct messages – host using 20 threads

Which gave these averaged results:

Boosting the number of threads dramatically improved the NSB performance, and removing the distributor halved the duration. I would expect that running the distributor on a separate node would yield similarly good results on each processing node. The hybrid BizTalk/NSB configurations were the slowest – to be expected, as you have the latency of the message box and the latency of MSMQ saving to disk. NSB configurations without recoverable messaging were only marginally slower than the recoverable version, which seems incorrect based on Udi Dahan’s performance benchmark, so I’ll need to look into that further.

There’s plenty of scope for improving performance in both solutions. BizTalk can be endlessly tuned (the 2004 guidelines still apply as a good starting point). By modifying the NSB solution to send the AddProduct messages in one opration and using 80 threads for the handler, duration fell to 25 seconds – 140 messages per second.

Integrating BizTalk and nServiceBus

Despite having the worst performance, integrating BizTalk and nServiceBus is a viable option which may be very useful in some cases. It would allow you to leverage BizTalk’s adapter suite and mapping functionality to join LOB systems into an nServiceBus estate. Integration is actually very straightforward. By default NSB uses MSMQ, so to publish messages from BizTalk to an NSB subscriber just means configuring an MSMQ send port with the expected queue, and mapping your message to NSB format – which is an envelope containing one or messages:

<?xml version="1.0" ?>

<BatchSourcePath>E:\ExcelUpload\1.0.0.0\Binaries\Drops\ProductUpload.xls</BatchSourcePath>

<OriginatorDestination>ExcelUpload.Client.InputQueue@WIN2003R2-VM</OriginatorDestination>

<Name>new product 58</Name>

</AddProduct>

</Messages>

The MSMQ adapter lets you specify Recoverable and Transactional flags, so your BizTalk-generated messages have the same durability options. When an NSB handler is listening at the queue, it processes the BizTalk messages in the same way as NSB messages.

Publishing NSB messages to BizTalk is trickier, as the Send and Publish methods from NSB check to see if there are any subscribers before they write to the queue (compare this to the Notify method in RhinoServiceBus). If a BizTalk MSMQ receive port is the only handler, no subscribers will be registered with NSB and the message won’t be published. So you need to either call Send with a named queue (Bus.Send<AddProduct>("ExcelUpload.AddProductService.1.InputQueue", m =>...), or send a subscription message from BizTalk to register a subscriber. In the the second option you’d need to specify the message type in the subscription:

<?xml version="1.0" ?>

<string>ExcelUpload.Messages.StartBatchUpload, ExcelUpload.Messages, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null</string>

Running the Sample

(If you haven’t seen the original NSB sample, have a read as it details the pre-requisites).

To run the BizTalk solution, download ExcelUpload.BizTalk.Binaries.zip, unzip it and:

· Copy ExcelUpload.PipelineComponents.dll to your pipeline components directory – e.g. C:\Program Files\Microsoft BizTalk Server 2006\Pipeline Components

· If you haven’t done so from the previous sample, run uspInsertProduct.CREATE.sql

· Run uspInsertProduct.CREATESYNONYM.sql

· Import the MSI into BizTalk

· Run the MSI

· Start send port SendInsertProduct.SQL (you’ll need to change the connection details)

· Enable receive location ReceiveStartUpload.FILE.XLS (monitors c:\drops\ExcelUpload)

· Drop an Excel file

As before, you’ll need to clear down the database between runs if you drop the same file repeatedly.

If you want to run the hybrid solutions, you should use the binaries in the new download rather than the original, as these run without a distributor. Run start.cmd and there are three windows – Client (the file watcher), Host (the StartBatchUpload handler) and AddProductService (this is the AddProduct handler).

To run configuration 6 – NSB parsing the file and BizTalk handling the AddProduct messages:

· Run start.cmd

· Kill the AddProductService console

· Disable receive location ReceiveStartUpload.FILE.XLS

· Enable receive location ReceiveAddProduct.MSMQ

· Unenlist send port SendAddProduct.MSMQ

· Start send port SendInsertProduct.SQL

· Drop an Excel file

To run configuration 7 – BizTalk parsing the file and NSB handling the AddProduct messages:

· Run start.cmd

· Kill the Host and Client consoles

· Enable receive location ReceiveStartUpload.FILE.XLS

· Disable receive location ReceiveAddProduct.MSMQ

· Start send port SendAddProduct.MSMQ

· Unenlist send port SendInsertProduct.SQL

· Drop an Excel file

Source Code

The source and referenced assemblies are in ExcelUpload.BizTalk.Source.zip. There is a VS 2008 solution for the nServiceBus components, and a VS 2005 solution for the BizTalk components.

Processing an Excel upload with nServiceBus

EltonStoneman — Wed, 18 Nov 2009 09:46:13 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

We’ve had a couple of projects recently with similar requirements to process an Excel file as a batch upload of data. One was a BizTalk project where the FarPoint Spread pipeline component was a good fit; the other was a Web app where we put together a custom parser based on the open-source ExcelDataReader. The custom solution was appropriate for the expected size of upload files, but wouldn’t scale well to deal with large files quickly, so I wanted to look at a distributed alternative using nServiceBus. My sample implementation is on MSDN Code Gallery here: nServiceBus Excel Upload. I’ll look at a comparative BizTalk solution in a future post.

If you haven’t come across nServiceBus, it’s a queue-based messaging framework which is inherently asynchronous. “Scalability and reliability are in its DNA”, and it has some impressive case studies. Using nServiceBus you can set up a simple publish-subscribe architecture between nodes, or a load-balanced architecture with a central distributor. In the distributed version, the upload sample looks like this:

(Note that the diagram represents the bus as a separate entity, in reality it’s distributed among the queues of all the nodes. The diagram also omits the distributor).

In nServiceBus, services are requested by publishing messages onto the bus. Requests are fulfilled by a handler which subscribes to a type of message. The Excel upload sample takes a workbook which contains a set of products and uploads them to the AdventureWorks database. There are three types of message:

StartBatchUpload – published when a file has been received and is ready to be processed; subscriber does some basic validation on the Excel data structure, and then for each row in the worksheet publishes an AddProduct message;
AddProduct – subscriber maps the product defined in the message to a stored procedure call which inserts the new product. When the last product in the batch is reached, sends a BatchStatusChanged message to the original publisher of the StartBatchUpload message;
BatchStatusChanged – logs the status change and renames the Excel upload file.

This is a basic example, more validation would be expected, but the workflow is representative. Parsing the Excel file is done quickly, allowing for any number of nodes to participate in the resource-intensive work of creating the products. Using a single host with 5 threads, an Excel file with 3,500 rows takes just over 4 minutes to process on a dev laptop. That’s 13 messages per second which is nothing special, but this is on a single host which is also running the distributor and SQL Server. The processing host has a flat memory profile (consistently around 40Mb) and runs at less than 20% CPU. The distributor takes around 15% CPU, and MSMQ another 15%.

For a much larger upload – 12,000 rows – the processing and memory profile is the same, and the upload takes around 14 minutes (~14 messages per second) on the same infrastructure.

Running the Sample

Access to a SQL Server instance with the AdventureWorks sample database installed is a pre-requisite. You’ll need to add the new stored procedure with uspInsertProduct.CREATE.sql. The connection string used by the host is specified in ExcelUpload.Host.exe.config (defaults to unnamed local instance).

You’ll need MSMQ running on all nodes. Queues are specified in configuration and are created by nServiceBus if they don’t exist – an exception is the storage queue for the distributor which needs to be manually created, this PowerShell snippet will do it:

[Reflection.Assembly]::LoadWithPartialName("System.Messaging")

[System.Messaging.MessageQueue]::Create(".\Private$\distributorStorage", $true)

Unzip the file ExcelUpload.Binaries.zip. You’ll have a batch file – start.cmd – and five subdirectories – Client, Host, Distributor, SampleFiles and Drops. Run start.cmd, check the console screens for errors, then copy one of the Excel files from SampleFiles to Drops. You should see activity in the host, client and distributor console screens, and new rows being added to the [Production].[Product] table.

If you drop the same file twice, the unique key on Products will be violated, so the upload will error. On a fresh install there are under 1000 products, so this resets the table to the default state:

delete [Production].[Product] where ProductID > 999;

Implementation Details

The sample uses the release version of nServiceBus – 1.9 – as the distributor was broken in the 2.0 beta at the time of writing.

The two console apps run the “client” (which monitors a configured file location for an Excel drop, and publishes the StartBatchUpload message), and the “host” (which subscribes to StartBatchUpload and publishes AddProduct and BatchStatusChanged messages). Both use Topshelf so they can run as a console, or can be installed as a Windows service (e.g. ExcelUpload.Client.exe /install).

If you want to run several hosts on the same machine, they will need to use different queues. Copy the whole of the Host directory, and modify ExcelUpload.Host.exe.config to specify a unique queue name:

InputQueue="ExcelUpload.Service.1.InputQueue"

Then run ExcelUpload.Host.exe from all the copied locations, and you’ll see the console hosts sharing the message processing when a file is dropped.