Elton Stoneman

Processing Excel uploads with BizTalk and nServiceBus

EltonStoneman — Sat, 21 Nov 2009 07:57:56 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

Following on from the sample for processing Excel uploads with nServiceBus, I have a comparable version using BizTalk on MSDN Code Gallery here: BizTalk and nServiceBus Excel Upload. The BizTalk (2006 R2) sample uses a FILE receive port with a simple pipeline component to disassemble the Excel file into separate messages. A SQL send port subscribes to the messages, and with an outbound map calls the AdventureWorks stored procedure to insert each product.

Processing Excel in BizTalk is nothing new, but I wanted to do a comparison against the nServiceBus example, and also see how BizTalk and nServiceBus could be integrated. My original thinking was that the trigger in the nServiceBus solution relies on a FileSystemWatcher, which is less reliable and less flexible than BizTalk’s FILE adapter. A hybrid solution could use the FILE adapter to receive and parse the upload, then send each row as an AddProduct message to MSMQ, which the nServiceBus handler subscribes to.

Running this on the same environment (a Windows Server 2003 VM running under VirtualBox in Ubuntu 9.04), the BizTalk solution processes the 3,500 row Excel file in 3 mins 10 seconds (compared to 4m 15s for the distributed NSB running with 5 threads), and the 12,000 row file in 11m 14s (compared to 14m 0s). I was surprised to find the BizTalk solution running more quickly, as the original NSB was using non-recoverable messaging, so all messages were in memory, while BizTalk had the latency of writing to the message box.

Performance Comparison

This is not an intended to be a thorough benchmark of NSB and BizTalk – both sample projects are basic, untuned implementations, and the tests are on a single box rather than two or three. But for comparison, I ran the 3,500 row upload repeatedly under different configurations:

1. NSB with distributor – host and distributor using 20 threads

2. NSB with distributor – host and distributor using 20 threads, recoverable messaging

3. NSB without distributor – host using 20 threads

4. NSB without distributor – host using 20 threads, recoverable messaging

5. BizTalk FILE receive and SQL send

6. NSB parsing Excel file, BizTalk subscribing to AddProduct messages – MSMQ receive using batches of 100

7. BizTalk parsing Excel file, NSB without distributor subscribing to AddProduct messages – host using 20 threads

Which gave these averaged results:

Boosting the number of threads dramatically improved the NSB performance, and removing the distributor halved the duration. I would expect that running the distributor on a separate node would yield similarly good results on each processing node. The hybrid BizTalk/NSB configurations were the slowest – to be expected, as you have the latency of the message box and the latency of MSMQ saving to disk. NSB configurations without recoverable messaging were only marginally slower than the recoverable version, which seems incorrect based on Udi Dahan’s performance benchmark, so I’ll need to look into that further.

There’s plenty of scope for improving performance in both solutions. BizTalk can be endlessly tuned (the 2004 guidelines still apply as a good starting point). By modifying the NSB solution to send the AddProduct messages in one opration and using 80 threads for the handler, duration fell to 25 seconds – 140 messages per second.

Integrating BizTalk and nServiceBus

Despite having the worst performance, integrating BizTalk and nServiceBus is a viable option which may be very useful in some cases. It would allow you to leverage BizTalk’s adapter suite and mapping functionality to join LOB systems into an nServiceBus estate. Integration is actually very straightforward. By default NSB uses MSMQ, so to publish messages from BizTalk to an NSB subscriber just means configuring an MSMQ send port with the expected queue, and mapping your message to NSB format – which is an envelope containing one or messages:

<?xml version="1.0" ?>

<BatchSourcePath>E:\ExcelUpload\1.0.0.0\Binaries\Drops\ProductUpload.xls</BatchSourcePath>

<OriginatorDestination>ExcelUpload.Client.InputQueue@WIN2003R2-VM</OriginatorDestination>

<Name>new product 58</Name>

</AddProduct>

</Messages>

The MSMQ adapter lets you specify Recoverable and Transactional flags, so your BizTalk-generated messages have the same durability options. When an NSB handler is listening at the queue, it processes the BizTalk messages in the same way as NSB messages.

Publishing NSB messages to BizTalk is trickier, as the Send and Publish methods from NSB check to see if there are any subscribers before they write to the queue (compare this to the Notify method in RhinoServiceBus). If a BizTalk MSMQ receive port is the only handler, no subscribers will be registered with NSB and the message won’t be published. So you need to either call Send with a named queue (Bus.Send<AddProduct>("ExcelUpload.AddProductService.1.InputQueue", m =>...), or send a subscription message from BizTalk to register a subscriber. In the the second option you’d need to specify the message type in the subscription:

<?xml version="1.0" ?>

<string>ExcelUpload.Messages.StartBatchUpload, ExcelUpload.Messages, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null</string>

Running the Sample

(If you haven’t seen the original NSB sample, have a read as it details the pre-requisites).

To run the BizTalk solution, download ExcelUpload.BizTalk.Binaries.zip, unzip it and:

· Copy ExcelUpload.PipelineComponents.dll to your pipeline components directory – e.g. C:\Program Files\Microsoft BizTalk Server 2006\Pipeline Components

· If you haven’t done so from the previous sample, run uspInsertProduct.CREATE.sql

· Run uspInsertProduct.CREATESYNONYM.sql

· Import the MSI into BizTalk

· Run the MSI

· Start send port SendInsertProduct.SQL (you’ll need to change the connection details)

· Enable receive location ReceiveStartUpload.FILE.XLS (monitors c:\drops\ExcelUpload)

· Drop an Excel file

As before, you’ll need to clear down the database between runs if you drop the same file repeatedly.

If you want to run the hybrid solutions, you should use the binaries in the new download rather than the original, as these run without a distributor. Run start.cmd and there are three windows – Client (the file watcher), Host (the StartBatchUpload handler) and AddProductService (this is the AddProduct handler).

To run configuration 6 – NSB parsing the file and BizTalk handling the AddProduct messages:

· Run start.cmd

· Kill the AddProductService console

· Disable receive location ReceiveStartUpload.FILE.XLS

· Enable receive location ReceiveAddProduct.MSMQ

· Unenlist send port SendAddProduct.MSMQ

· Start send port SendInsertProduct.SQL

· Drop an Excel file

To run configuration 7 – BizTalk parsing the file and NSB handling the AddProduct messages:

· Run start.cmd

· Kill the Host and Client consoles

· Enable receive location ReceiveStartUpload.FILE.XLS

· Disable receive location ReceiveAddProduct.MSMQ

· Start send port SendAddProduct.MSMQ

· Unenlist send port SendInsertProduct.SQL

· Drop an Excel file

Source Code

The source and referenced assemblies are in ExcelUpload.BizTalk.Source.zip. There is a VS 2008 solution for the nServiceBus components, and a VS 2005 solution for the BizTalk components.

Processing an Excel upload with nServiceBus

EltonStoneman — Wed, 18 Nov 2009 15:46:13 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

We’ve had a couple of projects recently with similar requirements to process an Excel file as a batch upload of data. One was a BizTalk project where the FarPoint Spread pipeline component was a good fit; the other was a Web app where we put together a custom parser based on the open-source ExcelDataReader. The custom solution was appropriate for the expected size of upload files, but wouldn’t scale well to deal with large files quickly, so I wanted to look at a distributed alternative using nServiceBus. My sample implementation is on MSDN Code Gallery here: nServiceBus Excel Upload. I’ll look at a comparative BizTalk solution in a future post.

If you haven’t come across nServiceBus, it’s a queue-based messaging framework which is inherently asynchronous. “Scalability and reliability are in its DNA”, and it has some impressive case studies. Using nServiceBus you can set up a simple publish-subscribe architecture between nodes, or a load-balanced architecture with a central distributor. In the distributed version, the upload sample looks like this:

(Note that the diagram represents the bus as a separate entity, in reality it’s distributed among the queues of all the nodes. The diagram also omits the distributor).

In nServiceBus, services are requested by publishing messages onto the bus. Requests are fulfilled by a handler which subscribes to a type of message. The Excel upload sample takes a workbook which contains a set of products and uploads them to the AdventureWorks database. There are three types of message:

StartBatchUpload – published when a file has been received and is ready to be processed; subscriber does some basic validation on the Excel data structure, and then for each row in the worksheet publishes an AddProduct message;
AddProduct – subscriber maps the product defined in the message to a stored procedure call which inserts the new product. When the last product in the batch is reached, sends a BatchStatusChanged message to the original publisher of the StartBatchUpload message;
BatchStatusChanged – logs the status change and renames the Excel upload file.

This is a basic example, more validation would be expected, but the workflow is representative. Parsing the Excel file is done quickly, allowing for any number of nodes to participate in the resource-intensive work of creating the products. Using a single host with 5 threads, an Excel file with 3,500 rows takes just over 4 minutes to process on a dev laptop. That’s 13 messages per second which is nothing special, but this is on a single host which is also running the distributor and SQL Server. The processing host has a flat memory profile (consistently around 40Mb) and runs at less than 20% CPU. The distributor takes around 15% CPU, and MSMQ another 15%.

For a much larger upload – 12,000 rows – the processing and memory profile is the same, and the upload takes around 14 minutes (~14 messages per second) on the same infrastructure.

Running the Sample

Access to a SQL Server instance with the AdventureWorks sample database installed is a pre-requisite. You’ll need to add the new stored procedure with uspInsertProduct.CREATE.sql. The connection string used by the host is specified in ExcelUpload.Host.exe.config (defaults to unnamed local instance).

You’ll need MSMQ running on all nodes. Queues are specified in configuration and are created by nServiceBus if they don’t exist – an exception is the storage queue for the distributor which needs to be manually created, this PowerShell snippet will do it:

[Reflection.Assembly]::LoadWithPartialName("System.Messaging")

[System.Messaging.MessageQueue]::Create(".\Private$\distributorStorage", $true)

Unzip the file ExcelUpload.Binaries.zip. You’ll have a batch file – start.cmd – and five subdirectories – Client, Host, Distributor, SampleFiles and Drops. Run start.cmd, check the console screens for errors, then copy one of the Excel files from SampleFiles to Drops. You should see activity in the host, client and distributor console screens, and new rows being added to the [Production].[Product] table.

If you drop the same file twice, the unique key on Products will be violated, so the upload will error. On a fresh install there are under 1000 products, so this resets the table to the default state:

delete [Production].[Product] where ProductID > 999;

Implementation Details

The sample uses the release version of nServiceBus – 1.9 – as the distributor was broken in the 2.0 beta at the time of writing.

The two console apps run the “client” (which monitors a configured file location for an Excel drop, and publishes the StartBatchUpload message), and the “host” (which subscribes to StartBatchUpload and publishes AddProduct and BatchStatusChanged messages). Both use Topshelf so they can run as a console, or can be installed as a Windows service (e.g. ExcelUpload.Client.exe /install).

If you want to run several hosts on the same machine, they will need to use different queues. Copy the whole of the Host directory, and modify ExcelUpload.Host.exe.config to specify a unique queue name:

InputQueue="ExcelUpload.Service.1.InputQueue"

Then run ExcelUpload.Host.exe from all the copied locations, and you’ll see the console hosts sharing the message processing when a file is dropped.

Retrieving Nested Properties from Lambda Expressions

EltonStoneman — Fri, 06 Nov 2009 04:14:55 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

Supposing you’re using a lambda expression as a neat way to access a named property:

s => s.FirstName

- when what you really want is to access it without using a string for the property name (“FirstName”), for example to specify the source or target property for a mapping exercise.

It’s easy if the expression is a direct property of the parameter type – a member of the “s” in the expression; casting the whole expression as a MemberExpression gives you access to the member as a PropertyInfo. You can encapsulate it in an extension method of LambdaExpression:

public static PropertyInfo AsPropertyInfo(this LambdaExpression expression)

{

PropertyInfo info = null;

if (expression.Body.NodeType == ExpressionType.MemberAccess)

{

info = ((MemberExpression)expression.Body).Member as PropertyInfo;

}

return info;

}

- which lets you access the property like this:

Expression<Func<FullUser, object>> exp = s => s.FirstName;

PropertyInfo sourceProperty = exp.AsPropertyInfo();

But if the expression represents a property of a nested type, or a property of a nested type of a nested type:

s => s.Address.Line1

s => s.Address.PostCode.Code

- then it’s different.

What you want in this case is an executable expression which will access the containing object of the property (e.g. get you the Address property of the source object, or the PostCode property of the Address property of the source object). You also want the PropertyInfo of the target property (e.g. Line1 or Code). And you want both the property accessor and the PropertyInfo from the same expression.

This is not too difficult. It doesn’t require a stack of target objects and PropertyInfos as originally feared. Firstly store the PropertyInfo from the nested target in the normal way:

targetPropertyInfo = targetExpression.AsPropertyInfo();

- this gives us access to Line1 or Code. Then store the expression which retrieves the path to the containing object for the property from the original source - .Address or .Address.Postcode. In the expression tree, this is the expression which contains the root to the original parameter:

if (targetExpression.Body.NodeType == ExpressionType.MemberAccess)

{

MemberExpression memberExpression = targetExpression.Body as MemberExpression;

if (memberExpression.Expression.NodeType != ExpressionType.Parameter)

{

ParameterExpression parameter = GetParameterExpression(memberExpression.Expression);

if (parameter != null)

{

_targetExpression = Expression.Lambda(memberExpression.Expression, parameter);

}

The key line is the one which contains Expression.Lambda(). The parent expression is a MemberExpression (e.g. s.Address, or s.Address.PostCode) which itself is not executable. You can’t compile a MemberExpression and execute it on an object to get the member, as the expression needs to know the parameter it executes against.

Expression.Lambda creates a compilable, executable expression against a provided target parameter – which we already have from the “s” of the original expression, and we can find for any level of nesting:

private ParameterExpression GetParameterExpression(Expression expression)

{

while (expression.NodeType == ExpressionType.MemberAccess)

{

expression = ((MemberExpression)expression).Expression;

}

if (expression.NodeType == ExpressionType.Parameter)

{

return (ParameterExpression)expression;

}

return null;

}

So now we have the PropertyInfo for the nested target, and an expression which will give us the nested target, we can set the target property value like this:

object target; //this is the top-level object

//...

realTarget = expression.Compile().DynamicInvoke(target);

targetPropertyInfo.SetValue(realTarget, value, null);

It’s quite a complex requirement, but the solution is straightforward and the complexity is isolated from the calling code. Note that the example code above only deals with nested properties, not expressions with mixed properties and method calls (e.g. s.Address.GetPostCode().InwardCode), but it’s straightforward to cover those cases too.

Avoiding TNSNAMES in Oracle .NET data connections

EltonStoneman — Fri, 06 Nov 2009 04:03:04 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

The typical approach for connecting to Oracle data sources using ODP.NET is to use a named source in the connection string and rely on the name being resolved through SQLNAMES.ORA and TNSNAMES.ORA:

app.config:

</connectionStrings>

TNSNAMES.ORA:

XYZ.WORLD =

(DESCRIPTION =

(ADDRESS_LIST =

(LOAD_BALANCE = off)

(FAILOVER = on)

(ADDRESS_LIST =

(LOAD_BALANCE = on)

(FAILOVER = on)

(ADDRESS = (PROTOCOL = TCP)(HOST = XYZ1-1.internal.xyz.org.uk)(PORT = 1522))

(ADDRESS = (PROTOCOL = TCP)(HOST = XYZ1-2.internal.xyz.org.uk)(PORT = 1522))

)

(ADDRESS_LIST =

(LOAD_BALANCE = on)

(FAILOVER = on)

(ADDRESS = (PROTOCOL = TCP)(HOST = XYZ2-1.internal.xyz.org.uk)(PORT = 1522))

(ADDRESS = (PROTOCOL = TCP)(HOST = XYZ2-2.internal.xyz.org.uk)(PORT = 1522))

)

(CONNECT_DATA =

(SERVICE_NAME = XYZ)

(FAILOVER_MODE =

(TYPE = select)

(METHOD = basic)

(RETRIES = 16)

(DELAY = 1)

)

This is fine until you upgrade your Oracle client, or have multiple instances – if the wrong TNSNAMES gets picked up you’ll get error ORA-12514: TNS:could not resolve the connect identifier specified, and it can be fiddly to track down.

A neater way is to put the whole set of service details in the connection string, avoid the named identifier and circumvent TNSNAMES.ORA altogether:

<add name="XYZ" connectionString="Data Source=(DESCRIPTION =

(ADDRESS_LIST =

(LOAD_BALANCE = off)

(FAILOVER = on)

(ADDRESS_LIST =

(LOAD_BALANCE = on)

(FAILOVER = on)

(ADDRESS = (PROTOCOL = TCP)(HOST = XYZ1-1.internal.xyz.org.uk)(PORT = 1522))

(ADDRESS = (PROTOCOL = TCP)(HOST = XYZ1-2.internal.xyz.org.uk)(PORT = 1522))

)

(ADDRESS_LIST =

(LOAD_BALANCE = on)

(FAILOVER = on)

(ADDRESS = (PROTOCOL = TCP)(HOST = XYZ2-1.internal.xyz.org.uk)(PORT = 1522))

)

(CONNECT_DATA =

(SERVICE_NAME = UVASA)

(FAILOVER_MODE =

(TYPE = select)

(METHOD = basic)

(RETRIES = 16)

(DELAY = 1)

)

);User ID=xyz;Password=xyz;"/>

</connectionStrings>

Service details contain the whole of the TNSNAMES entry following the identifier (i.e. from DESCRIPTION= onwards), and can span multiple lines in the config file.

Incidentally, if you’re not using ODP.NET in favour of the Microsoft’s framework Oracle provider, System.Data.OracleClient, consider migrating as it will be deprecated from .NET 4.0.

jQuery Date Validation in Chrome

EltonStoneman — Thu, 29 Oct 2009 19:20:53 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

We had a fiddly issue with date validation in an ASP.NET MVC page failing for a valid date in Chrome, but passing in Firefox, IE etc. Tracing through our own code and xVal, the issue was narrowed down to the jQuery validation plugin (jquery.validate.js). For simple date validation, the library instantiates a date object from the given text value and lets JavaScript raise errors for invalid dates:

date: function(value, element) {

return this.optional(element) || !/Invalid|NaN/.test(new Date(value));

Chrome seems to ignore the current locale when it creates the date object, so a valid date in UK dd/mm/yy format – 29/10/2009 – fails as Chrome seems to interpret it in US mm/dd/yy format. Easy to test outside of all other code by entering some simple JavaScript into the address bar:

javascript:new Date('29/10/2009').toString()

Which Firefox renders as expected:

- as does IE (even IE6):

- but which Chrome renders as an invalid date:

You can also test it by navigating to the jQuery date validation sample page, and checking results for the same date in different browsers.

The fix is a simple change to the jQuery script to force the use of the current locale:

// http://docs.jquery.com/Plugins/Validation/Methods/date

date: function(value, element) {

//ES - Chrome does not use the locale when new Date objects instantiated:

//return this.optional(element) || !/Invalid|NaN/.test(new Date(value));

var d = new Date();

return this.optional(element) || !/Invalid|NaN/.test(new Date(d.toLocaleDateString(value)));

Duck Typing with Castle

EltonStoneman — Wed, 21 Oct 2009 07:59:24 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

David Meyer has a neat duck typing library for .NET which I’ve been looking at as I had a requirement to treat objects of various types in the same way. They all expose a “Value” property, but don’t implement an interface.

The library does more than I needed so I put together a quick duck typing implementation which just lets me treat an instance of an object which tacitly implements an interface, as a true implementation of the interface. In this example I can access BespokeDateTime and BespokeInt as IHasValue:

Using Castle’s Dynamic Proxy, this is simple to do – just requiring you to generate an interface proxy with a custom interceptor. The interceptor invokes the interface method on the real object, so you effectively wrap an instance in an interface it doesn’t explicitly implement:

BespokeDateTime bespoke = new BespokeDateTime(new DateTime(2000, 1, 1));

IHasValue duck = DuckType.As<IHasValue>(bespoke);

Console.WriteLine(duck.Value);

- or using an object extension to hide the implementation:

bespoke.As<IHasValue>().Value;

Iterating over collections is much cleaner with duck typing – if I have a collection that contains both BespokeDateTime and BespokeInt objects, to get all their values I can use:

foreach (object obj in GetObjects())

{

Assert.IsNotNull(DuckType.As<IHasValue>(obj).Value);

}

- compared to casting each object:

foreach (object obj in GetObjects())

{

BespokeDateTime bespokeDateTime = obj as BespokeDateTime;

if (bespokeDateTime != null)

{

Assert.IsNotNull(bespokeDateTime.Value);

}

else

{

BespokeInt bespokeInt = (BespokeInt) obj;

Assert.IsNotNull(bespokeInt.Value);

}

Use with caution over large collections, as there is a noticeable performance hit in using the duck type. Over 500,000 items the duck-typed code takes 40-50 times as long as the casting code, averaging 8 seconds to 0.2 seconds, with duck-typing by extension taking marginally longer again:

The full code (all 34 lines – including validation that the type to duck does tacitly implement the interface) is on MSDN Code Gallery here: Duck Typing with Castle.

The People’s Jukebox

EltonStoneman — Wed, 30 Sep 2009 19:00:38 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

For a forthcoming celebration, I've been working on a jukebox web application:

The client controls the music being played on the server, so its intended for local networks where you want shared control of a central music player. If you can find a use for it, help yourself – it's on CodePlex here: The People's Jukebox. If nothing else it's a straightforward example of a Silverlight 2.0 client talking to WCF REST services, in what's probably a familiar domain.

The user experience is meant to be zero-guidance. You can try it out on a version where the UI functions but doesn't play any music: The People's Jukebox on sixeyed.com.

The initial screen shows what's currently playing and has a big button for you to select a track. In the next screen you choose either to search for a specific track, browse through the collection, or have the jukebox randomly choose some options for you. From any of those routes, you select a track, confirm your choice and it gets queued:

The queuing is what gives The People's Jukebox its democratic flavour – if you select a track which is already queued, you increment its vote count. When the current track finishes, the jukebox picks the next track with the most votes. When a track is selected the UI forces you to wait for a few seconds before you can continue, to discourage The People from abusing their power and making the same selection over and over again.

If no one's choosing tracks, the jukebox will randomly pick one which it hasn't already played, so it should keep playing until all the songs are played, or the web server is stopped.

Usage

Run the MSI and the jukebox Web site and services will be installed with default configuration options, using the installing user's "My Music" as the root folder to look for tracks. It will populate the track catalogue from the file system, expecting to find a conventional root/artist/album/track hierarchy:

My Music
- The Beatles
  - Abbey Road
    - 01. Come Together
    - etc.

Open http://localhost/ThePeoplesJukebox/ and hit Start - the music should start playing and you should be able to navigate around your music collection. The jukebox services log everything they do, by default to C:\TEMP\PeoplesJukeboxService.log, so if nothing happens that's the place to look for clues.

Configuration

The jukebox is configurable through Web.config on the services. Options are documented in comments – the installer sets up the root music directory, and the types of music files in the fileSystemConfiguration element:

<!--Specify the root path for music files, and

the types of files to load. Directory can be

a network path, extensions are comma-separated:-->

<fileSystemCatalogueConfiguration

rootDirectory="x:\My Music"

fileExtensions="*.wav,*.mp3"

rootDirectory can be local or a UNC path, e.g. \\nas\music.

Track Lists

Track lists are what drives the song selection behaviour; this is also configurable. Three types of tracklist are provided. When the jukebox finishes playing a track, it will query the tracklists in priority order until one returns a track, which will be loaded and played next:

<!-- Tracklists to use, and the order in which they are queried.

Specify "chosenByText" to display which tracklist chose the

Now Playing track on the client -->

</trackLists>

In default form, the jukebox lets you start off with some specified songs, moving onto the selections of The People, with the jukebox picking a random track if none are queued, so tracklists are queried in the following order:

1. ConfiguredTrackList – reads a list of specific track names from configuration, and returns them in priority order. Will not play a track that's already been played. Configuration looks like:

</tracks>

</trackConfiguration>

Intended to start the evening off with a known selection, when the configured tracks have been played, the list returns null so the jukebox moves on to the next tracklist.

2. SelectedTrackList – hopefully the main provider, queues up track requests as users make a selection. Operates the voting mechanism, so tracks which are selected multiple times move up the queue. If all queued tracks have the same number of votes, they're played in the order they were first selected. Will allow tracks to be played multiple times, but not sequentially – if you choose a track that's already queued, you increase its vote count; if you choose the same track again whilst its playing, or after it has been played, it gets queued again. Returns null if no tracks are queued.

3. RandomTrackList – failsafe list, to ensure music is always playing, picks tracks at random from the catalogue. Will not play a track that's already been played. Allows the administrator to sneak in some favourites by configuring "must-have" tracks. Must-haves will be inserted between random selections, at a configured frequency:

</mustHaveTracks>

</randomTrackListConfiguration>

- meaning every fifth track will be picked from the configured list, then the next four will be random. If no must-haves are specified, or they have all been played, the selection will be random.

Technical Details

Much of the implementation is straightforward. WCF has in-built support for REST, so you can expose a WCF service with a RESTful endpoint like http://localhost/PeoplesJukebox.Services/CatalogueService.svc/artist/2 (to get the list of tracks from artist with ID=2 from the catalogue service), just by flagging the service implementation with the WebGet attribute:

[WebGet(UriTemplate = "artist/{artistId}", ResponseFormat = WebMessageFormat.Xml)]

public Artist GetArtist(string artistId)

{...

Task-driven workflow implementation in Silverlight is also straightforward, by hosting multiple pages on the application, all of which are invisible except the current page. When the current page completes, it hides itself and loads the next page – handled through events in Page.xaml.cs:

public Page()

{

InitializeComponent();

foreach (PageBase page in Pages)

{

page.Complete += new PageBase.OnCompleteEventHandler(PageComplete);

page.Cancelled += new PageBase.OnCancelledEventHandler(PageCancelled);

}

this.CurrentPage = this.ucStartPage;

}

Silverlight 2.0 doesn't work so well with non-GET web requests, which is why all the services are GETs where some should really be POSTs.

The only difficult part was playing music files through IIS in a background thread, while continuing to respond to web service and web site requests. If you're really interested in looking into this, you'll see the music playing component is also specified in config:

<!-- Core configuration - which implementations to use for

media player and catalogue components -->

<peoplesJukeboxConfiguration

mediaPlayer="PeoplesJukebox.Core.Players.FMODPlayer, PeoplesJukebox.Core"

mediaCatalogue="PeoplesJukebox.Core.Catalogues.FileSystemCatalogue, PeoplesJukebox.Core"

There are several (failed) implementations in the source which are not built in the solution (under ..\PeoplesJukebox\PeoplesJukebox.Core\Players). Of the five attempts, all of them had showstopper issues except the excellent FMOD library:

DirectX – using the Audio class from the managed wrapper in the DirectX SDK. Redistributable install is massive, which is offputting, but more importantly the Audio.Ending event didn't fire reliably, so playback would intermittently stop and never resume;
Windows Media Player – using the managed WMPLib wrapper in the Windows Media Player SDK, to devolve playing responsibility to Media Player. Didn't like the extra dependency, or the alarming memory profile. Worse, the player would randomly change state to Stopping in the middle of playback and not resume;
Sound – using .NET's native sound player in System.Media. Fine for playing WAVs, at the cost of having to manage your own threads. Main issue that WAVs are not normalized so tracks will be played at their native volume, which is likely to mean volume changes between songs. Unable to play other media types;
Alvas Audio – looked good, but highly unfriendly API. Couldn't get the thing to play at all;
FMOD – highly-featured C++ audio processor which does its own thread management, copes with various media types, plays reliably for extended periods and has an efficient memory model. The library is free for non-commercial use, so The People's Jukebox has the same restriction. The library is complex, but I've used it through FMOD.NET – Rafael Nicoletti's very nice C# wrapper.

The mediaCatalogue element lets you choose how the catalogue is built. Currently, only a file system catalogue is provided, but extending it (e.g. to use SqueezeCenter – now SqueezeBox Server), is an option.

If you download the jukebox and find any issues using it, let me know through a comment or a CodePlex bug.

Resolving Special Folders in WiX 3.0

EltonStoneman — Tue, 29 Sep 2009 18:09:43 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

One of the frustrations of working with WiX is that when you want to do something out of the ordinary, chances are it's provided somewhere in a WiX extension, but unless you trawl through the XSDs you're not likely to find it. I had this issue when I wanted to resolve the special folder My Music through WiX, so I could put the full path into a config file.

Various special folders are already available as properties to the installer, including My Documents and My Pictures, but not My Music. Adding an extension to expose Environment.GetSpecialFolder() to WiX is a trivial implementation in a couple of classes (comments omitted for brevity):

public class EnvironmentExtension : WiXExtension

{

private EnvironmentPreprocessorExtension preprocessorExtension;

public override PreprocessorExtension PreprocessorExtension

{

get

{

if (this.preprocessorExtension == null)

{

this.preprocessorExtension = new EnvironmentPreprocessorExtension();

}

return this.preprocessorExtension;

}

public class EnvironmentPreprocessorExtension : PreprocessorExtension

{

private const string SPECIALFOLDER = "specialFolder";

public override string[] Prefixes { get { return new string[] { SPECIALFOLDER }; } }

private List<string> _specialFolders;

private List<string> SpecialFolders

{

get

{

if (_specialFolders == null)

{

_specialFolders = new List<string>(Enum.GetNames(typeof(Environment.SpecialFolder)));

}

return _specialFolders;

}

public override string GetVariableValue(string prefix, string name)

{

string result = null;

if (prefix == SPECIALFOLDER && SpecialFolders.Contains(name))

{

Environment.SpecialFolder specialFolder = (Environment.SpecialFolder)Enum.Parse(typeof(Environment.SpecialFolder), name, true);

result = Environment.GetFolderPath(specialFolder);

}

return result;

}

Reference the folder you want from the Environment.SpecialFolders enum as a property in your WiX script:

- and add the extension assembly to your light and candle calls, making sure the DLL is available at runtime :

candle -out x.y.z.WiXobj x.y.z.wxs -ext EnvironmentExtension

light -out x.y.z.msi x.y.z.WiXobj -ext EnvironmentExtension -cultures:en-us

But I suspect it's already available somewhere in WiX.

Lazy-Loading with Object Extensions

EltonStoneman — Wed, 23 Sep 2009 18:38:57 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

If you make heavy use of lazy-loaded properties, the repetitive if-backing-field-null-then-instantiate code can be centralized in an extension method on the object class:

/// <summary>

/// Extension methods for <see cref="Object"/>

/// </summary>

public static class ObjectExtensions

{

/// <summary>

/// Lazy-loads a property

/// </summary>

/// <typeparam name="T">Type of property</typeparam>

/// <param name="obj">Extended object</param>

/// <param name="field">Property backing field</param>

/// <param name="load">Property loading function</param>

/// <returns>Lazy-loaded property</returns>

public static T LazyLoad<T>(this object obj, ref T field, Func<T> load)

{

if (field == null || field.Equals(default(T)))

{

field = load();

}

return field;

}

Classes then define lazy-loaded properties using the extension method, specifying the backing field and the method used to populate it:

protected List<MediaItem> AllItems

{

get

{

return this.LazyLoad<List<MediaItem>>(ref this._allItems,

() => new List<MediaItem>());

}

- rather than:

protected List<MediaItem> AllItems

{

get

{

if (this._allItems == null)

{

this._allItems = new List<MediaItem>();

}

return this._allItems;

}

Only a tiny enhancement, but it isolates some general code and reduces the chances of causing an infinite loop by typing the property name rather than the field name.

CA1502:AvoidExcessiveComplexity and Lambdas

EltonStoneman — Mon, 21 Sep 2009 20:47:01 GMT

[Source: http://geekswithblogs.net/EltonStoneman]

In code with lots of lambda expressions, you may trigger a false positive for the Code Analysis rule CA1502 "AvoidExcessiveComplexity". The rule calculates cyclomatic complexity using the logic in Microsoft.FxCop.Sdk.MethodMetrics, which enumerates the instruction set and increments complexity count when it finds one of a given set of opcodes.

The rule was triggered by code which uses the Fluent DAL approach to populate a composite entity:

public static Entity GetEntity(int entityId)

{

//populate basic details:

Entity entity = Fluently.Load<Entity>().With<EntityMap>()

.From<GetEntity>(i => i.Id = entityId,

x => x.Execute());

//load child entities:

if (entity.IsKnown)

{

entity.Address = Fluently.Load<Address>().With<AddressMap>()

.From<GetEntityAddresses>(i => i.Id = entityId,

x => x.Execute());

entity.Contact = Fluently.Load<Contact>().With<ContactMap>()

.From<GetEntityContacts>(i => i.Id = entityId,

x => x.Execute());

}

else

{

//more of the same...

The method in question had a single if/else branch, no other conditional logic, but 7 fluent calls totalling 12 lambdas. I calculate the cyclomatic complexity to be 2, and NDepend agrees. FxCop tells me the complexity is 26.

I stripped this down to a much simpler sample and found the same issue:

public void CCCheck2()

{

Fluently.Load().With().From(i => i.ToString(), x => x.ToString());

}

- this gives a CC of 13, although it has no conditional logic at all.

The answer's in the IL – the lambdas are emitted as private static classes and executed using an instance which is lazy-loaded:

IL_000b: ldsfld class [System.Core]System.Func`2<int32,string> FxCopCC.LambdaSample::'CS$<>9__CachedAnonymousMethodDelegate6'

IL_0010: brtrue.s IL_0025

IL_0012: ldnull

IL_0013: ldftn string FxCopCC.LambdaSample::'<CCCheck>b__0'(int32)

IL_0019: newobj instance void class [System.Core]System.Func`2<int32,string>::.ctor(object,

native int)

IL_001e: stsfld class [System.Core]System.Func`2<int32,string> FxCopCC.LambdaSample::'CS$<>9__CachedAnonymousMethodDelegate6'

IL_0023: br.s IL_0025

br and brtrue are transfer of control statements, so every lambda expression actually causes a branch – where the IL checks to see if there's a cached instance of the delegate, or whether it needs to instantiate one – and a return. The complexity rating is correct for the IL, as 1 + 2x(number of lambdas).

But cyclomatic complexity is a measure of readability, so the additional branches in the IL should be irrelevant. The short term solution is to suppress the message on methods with heavy lambda usage. A longer term option is an alternative rule which uses a different calculation.

A side point is that the IL isn't aggressively optimised by the compiler – lambdas with identical code are generated as separate private classes, each with their own lazy-loading check, so to avoid that overhead lambdas which are reused can be centralised:

private Func<int, string> GetInitialise()

{

return new Func<int,string>(i => i.ToString());

}