Backing up SQL Azure

That's it!!! After many days and nights... and an amazing set of challenges, I just released the Enzo Backup for SQL Azure BETA product (http://www.bluesyntax.net). Clearly, that was one of the most challenging projects I have done so far.

Why???

Because to create a highly redundant system, expecting failures at all times for an operation that could take anywhere from a couple of minutes to a couple of hours, and still making sure that the operation completes at some point was remarkably challenging. Some routines have more error trapping that actual code...

Here are a few things I had to take into account:

  • Exponential Backoff (explained in another post)
  • Dual dynamic determination of number of rows to backup 
  • Dynamic reduction of batch rows used to restore the data
  • Implementation of a flexible BULK Insert API that the tool could use
  • Implementation of a custom Storage REST API to handle automatic retries
  • Automatic data chunking based on blob sizes
  • Compression of data
  • Implementation of the Task Parallel Library at multiple levels including deserialization of Azure Table rows and backup/restore operations
  • Full or Partial Restore operations
  • Implementation of a Ghost class to serialize/deserialize data tables

And that's just a partial list... I will explain what some of those mean in future blob posts. A lot of the complexities had to do with implementing a form of retry logic, depending on the resource and the operation.

 

PASS Conference 2011 Topic: Multitenant Design and Sharding with SQL Azure

I am really happy to announce that I have been accepted as a speaker at the 2011 PASS Conference in Seattle. The topic? It will be about SQL Azure scalability using shards, and the Data Federation feature of SQL Azure. I will also talk extensively about the community open-source sharding library Enzo SQL Shard (enzosqlshard.codeplex.com) and show how to make the most out of it.

In general, the presentation will provide details about how to properly design an application for sharding, how to make it work for SQL Server, SQL Azure, and how to leverage the upcoming Data Federation technology that Microsoft is planning. The primary objective is to turn sharding an implementation concern, not a development concern. Using a library like Enzo SQL Shard will help you achieve this objective.

If you come to PASS Summit this year, come see me and mention you saw this blog!

Twitter