Database Considerations

How To Search Encrypted Text in SQL Server 2005/2008

Chris Falter — Mon, 06 Oct 2008 05:14:56 GMT

Recently I was discussing SQL Server encryption with some friends who have been using it to encrypt short strings such as Social Security numbers at their shop. I commented, "Just try searching those Social Security Numbers," they shared my lamentation, and we moved on to other subjects. Later that evening, though, I thought there must be a way to search those wretched encrypted blocks--somehow--and worked out the solution you are about to read.

The Problem

The difficulty lies in the fact that you cannot compare varbinaries in a predicate. Combine this with the fact that encrypted text is stored as a varbinary, and the result is that you cannot search the varbinary encrypted text, even if you run the search term through the encryption algorithm before performing the comparison.

To illustrate, suppose you have a Person table that stores a name and a Social Security Number (SSN) encrypted with a certificate, defined like this:

CREATE TABLE [dbo].[Person](
[Id] [int] IDENTITY(1,1) NOT NULL,
[SSN] [varbinary](128) NULL,
[Name] [varchar](50) NULL,
CONSTRAINT [PK_Person] PRIMARY KEY CLUSTERED
(
[Id] ASC
)ON [PRIMARY]
)ON [PRIMARY]

It would be nice if you could just encrypt the search text and build a predicate that compares the term and the encrypted column, like this:

declare @encryptedSSN varbinary(128);
set @encryptedSSN = ENCRYPTBYCERT(CERT_ID('certKey'), @ssn);
select name, CONVERT(char(9),DECRYPTBYCERT(CERT_ID('certKey'), SSN) as SSN
from dbo.Person
where ssn = @encryptedSSN;

As I mentioned previously, though, this approach simply does not work.

Of course, you could make the predicate succeed by setting the datatype of the encrypted text column to char(128), and then converting the encryption products (stored SSNs plus the search term) from varbinary to char(128) before using them. However, indexing a column that large, and performing comparisons with it, would consume a lot of disk and processing resources.

The Solution

Instead of converting the large encrypted block into char(128) data, you can work with the hash of the SSN instead. If you calculate the hash using the SHA1 algorithm, the hash will require only 20 bytes of storage, so the burden on disk and processing resources will drop significantly. And since the SHA1 algorithm is considered to be very secure, you will not add any significant security risk to your solution. From start to finish, here's how the approach will work:

1. Include a char(20) column in your table to hold the hash of the SSN. The new table definition will look like this:

CREATE TABLE [dbo].[Person](
[Id] [int] IDENTITY(1,1) NOT NULL,
[SSN] [varbinary](128) NULL,
[Name] [varchar](50) NULL,
[Hash] [char](20) NULL,
CONSTRAINT [PK_Person] PRIMARY KEY CLUSTERED
(
[Id] ASC
)ON [PRIMARY]
)ON [PRIMARY]

You should also index the Hash column to improve the performance of searches by SSN, of course.

2. Whenever you insert a record, store the hash of the SSN along with the encrypted bytes. Your insert stored procedure should use the built-in HASHBYTES function, like this:

CREATE PROCEDURE [dbo].[usp_Insert_Person]
@ssn char(9),
@name varchar(50)
AS
BEGIN
insert into dbo.Person(SSN, Name,Hash)
values(ENCRYPTBYCERT(CERT_ID('certKey'), @ssn), @name, convert(char(20), HASHBYTES('SHA1', @ssn)))
END

3. When you search by SSN, compare the search term hash to the stored hash in your predicate:

CREATE PROCEDURE usp_Search_Person_BySsn
@ssn char(9)
AS
BEGIN
--SET NOCOUNT ON added to prevent extra result sets from
--interfering with SELECT statements.
SET NOCOUNT ON;

declare @hash char(20);
set @hash = convert(char(20), HASHBYTES('SHA1', @ssn));
select convert(char(9), DECRYPTBYCERT(cert_id('certkey'), ssn)) as SSN, name
from dbo.Person
where Hash= @hash;
END

Conclusion

This approach has worked well for me in my experimentation, and I recommend it to you if you need to search on an encrypted field. You should note, however, that the approach does have a limitation: you cannot search on an arbitrary substring of the encrypted text. In other words, a "begins with" or "contains" search is not possible. As long as you can live with this limitation, you should be able to employ this approach for your own encrypted data.

As always, I invite you, my reader, to leave a comment if you have found this useful or if you can think of an improvement.

.NET Entity Framework: Is It Ready for Web Prime-Time?

Chris Falter — Tue, 27 May 2008 14:50:29 GMT

Our shop has been looking for a way to simplify the interaction between our domain layer and data layer. The data layer uses typed datasets for all its operations. When it retrieves data, it populates a dataset; when it stores data, it checks the DataRowState of all rows in the dataset, and calls the appropriate stored procedure to insert, update, or delete the data. The domain layer then uses dataset elements as the in-memory backing store for properties and collections. A collection typically uses a DataTable, for example, and a collection member would use a DataRow in the DataTable.

This arrangement has some advantages:

Between requests, domain objects can cache their state in session by caching a dataset.
DataRowState can inform the logic about which CRUD operation to perform on a DataRow. Unmodified data can also be skipped over, yielding improved efficiency.
We have considerable freedom in designing domain classes. We can make a property read-only, for example, if it corresponds to an immutable database field, such as an identity column.

This arrangement also has a big disadvantage, however. We end up with lots of properties that look like this:

public string Phone

{

    get { return dr.PHONE; }

    set { dr.PHONE = value; }

}

This property's one and only use is to map a domain attribute (an entity's phone number) to a particular field in a DataRow (dr). Our domain layer has hundreds of these pass-through properties, which are a very verbose and not especially readable way of mapping between the domain and data layers. So we've been looking for a better way to handle this mapping.

Along comes the .NET Entity Framework (EF), which provides a tool that creates a domain-to-data map in XML format. Our first reaction was: this is just what the doctor ordered! And supported by Microsoft, to boot.

On closer examination, though, the EF toolset has some growing up to do before we could consider using it in our web applications. It can generate only public get/set accessors in the entity domain model (EDM) classes, for example. You cannot make a property read-only, or internal/protected/private, or virtual. You cannot map a non-generated property to a database field. And programming in a straitjacket was not what we had in mind for a new approach.

Second, EDM classes generated by EF do not have any supported way to cache their state. A Microsoftie is working on a tool ("Perseus") that can serialize the state of an entity using an EntityBag<T> instance, but it seems somewhat incomplete (and definitely not supported) at the moment. Since our web apps do cache domain entity state, adopting the EF right now would entail an important risk.

What alternatives exist? We can license DevForce EF, which is built on top of the EF and seems to offer pretty much everything that it lacks. Or we can try NHibernate, an open-source .NET port of the Hibernate tool which is quite popular in the Java development world.

Or maybe we can just wait for the EF to mature. Microsoft has big plans for the EF, and intends to make REST-oriented web services, Reporting Services, workflows, etc. EDM-aware. So an EF investment today can yield important dividends in the future.

Even though I cannot render a definitive answer, I hope that this discussion will help readers make an informed discussion about their ORM choices in the .NET world. What do you think? Leave a comment with your insights or questions!

How To: Encrypt and Manage Documents with SQL Server 2005

Chris Falter — Thu, 08 May 2008 21:13:56 GMT

When one of our departments decided to store sensitive reports, the architecture I first considered was storing them, unencrypted, in our imaging system. The imaging system cannot manage encrypted documents, but its security capabilities allow it to to deny unauthorized users access to the reports. This ability to restrict access seemed to meet our security requirements.

Then I started asking questions about data backups. Offsite backups can be a major security vulnerability, as we have recently learned. The names, addresses, social insurance numbers, and (in many cases) bank account details of almost half of England ended up who-knows-where last year because database backups disappeared in the postal system. And our company, like any responsible organization, backs up our data and stores it offsite. So leaving the sensitive data unencrypted was not acceptable.

Fortunately, SQL Server 2005 comes with native encryption capabilities that are simple to use. In the absence of an affordable imaging system with built-in encryption capabilities, here's the approach I hammered out:

Create a table in a SQL Server 2005 database to store the encrypted documents.
Create a symmetric key in the same database. To facilitate recovery from database failures, store the instructions for re-creating the key in a safe, remote location.
Create stored procedures that use the key to encrypt/decrypt documents that are stored in the table.
Create a service account under which the service that stores the report will run.
Create a (presumably internal) website for displaying the reports.
- Use the same service account for the site's app pool identity.
- Restrict access to authorized groups via Windows authentication.
- Require secure http so that a network trace cannot intercept decrypted documents on their way to the browser.
On SQL Server, deny permissions for the stored procedures and table to anyone other than the service account. The security context of the backup job must also be granted the db_backupoperator role on the database, of course.
Implement network encryption between SQL Server and any services that insert or select sensitive documents, using IPSec or SQL Server Network Encryption.

Below is a diagram of the system that allows authorized users to view sensitive documents.

There is scant MSDN documentation for accomplishing the first 3 steps, but the rest are well understood in the world of Microsoft development. The remainder of this article will therefore focus on creating and managing the tables, keys, and stored procedures.

Step #1: Create a Table to Store the Encrypted Documents

Several considerations drive the design of the table:

The column that stores a block of encrypted data must have the datatype of varbinary(8000) because the result of an encryption operation is always a varbinary(8000).
Unless you are dealing with very small documents, 8000 bytes will not hold an entire document. Therefore a single document will require multiple records, which will be associated with one another by a common unique identifier. A foreign key to the business entity associated with the document would be a good candidate. In our system, the encrypted document is associated with an "Order," so the identifier is an OrderId.
In order to thwart brute force decryption attacks, you should "salt" the encryption with a unique integer for each report. Unless your sensitive documents are very large, the foreign key (OrderId) will serve the purpose nicely.

Here's the table definition that emerges from these considerations:

CREATE TABLE [dbo].[EncryptedReport](
   [OrderId] [int] NOT NULL,
   [BlockNum] [int] NOT NULL,
   [Block] [varbinary](8000) NULL,
CONSTRAINT [PK_EncryptedReport] PRIMARY KEY CLUSTERED
(
   [OrderId] ASC,
   [BlockNum] ASC
)
) ON [PRIMARY]

Step #2: Create a Symmetric Key

To encrypt data in SQL Server 2005, you may use a certificate or a symmetric key. A certificate provides somewhat stronger protection than a symmetric key, so it is the encryption method of choice for a small amount of data (such as a Social Security Number). However, its far greater demand for server resources makes it impractical for large amounts of text or other data. Therefore, if you are encrypting a sensitive document, you should create a symmetric key and encrypt it with a certificate. At run-time, SQL Server will use the certificate to decrypt the key, then use the key to encrypt or decrypt the document.

Before SQL Server can encrypt or decrypt a sensitive document for you, it must traverse a chain of encrypted secrets:

A certificate available only to the SQL Server process identity encrypts SQL Server's Service Master Key.
The Service Master Key encrypts the Database Master Key.
The Database Master key encrypts a certificate ("certDocEncryption" in this example).
The certificate encrypts a symmetric key ("keyReportEncryption" in this example).
(Finally!) the symmetric key encrypts your sensitive documents.

SQL Server stores all of these secrets for you, except for the process identity's certificate (which is stored in the Windows certificate store). Consequently, you do not need to worry about making a "secret" available to your run-time environment by, for example, storing it in a configuration file. Database backups will not include the process identity's certificate (which the operating system maintains for you), so the only way the bad guy that steals your backup can decrypt your sensitive documents is by a brute force attack.

This design does place certain responsibilities on your IT organization:

You must be very careful about using this design with a virtual server. While creating an image of an entire virtual server (Windows OS, SQL Server, and data) facilitates quick disaster recovery, it can also facilitate the theft of sensitive documents, since the virtual server image will have everything (from Windows certificate down to the symmetric key) that a hacker needs to read the data.
To recover data in the event of a server failure, you must be able to re-create the symmetric key. While it is impossible to back up a symmetric key, you can always create an identical key on any instance of SQL Server 2005 by issuing the "CREATE SYMMETRIC KEY" statement with the same algorithm, key source, and identity value. Consequently, you must store the statement used to create your symmetric key in a secure location, such as a safe deposit box at a local bank. And although this statement is quite obvious, I'll say it anyway: don't store the statement with your data backups!

Step #3: Create Stored Procedures

Encrypting a sensitive document is a four-step process:

Open the symmetric key.
Break the document into blocks that, when encrypted, will consume less than 8000 bytes. When I applied Microsoft's published formula to my encryption key process, I concluded that 7876 bytes was the maximum size of unencrypted data that I could safely fit into a varbinary(8000).
Encrypt each block.
Insert each encrypted block into the table.

Here's the definition of the stored procedure that does the trick:

CREATE PROCEDURE [dbo].[usp_StoreReportWithEncryption] 

    @OrderId int, 

    @ReportText varchar(max) = ''

AS

BEGIN

    declare @idx int

    declare @textLength int

    declare @blockSize int

    declare @blockNum int

    declare @block varbinary(8000)

    declare @keyGuid uniqueidentifier

    Open Symmetric Key keyReportEncryption

        decryption by certificate certDocEncryption

    set @keyGuid = Key_GUID('keyReportEncryption')

    set @blockSize = 7876

    set @blockNum = 1

    set @idx = 1

    set @textLength = datalength(@ReportText)

    BEGIN

        WHILE @idx < @textLength

        BEGIN

            set @block = EncryptByKey(@keyGuid, SUBSTRING(@ReportText, @idx, @blockSize), 1, Convert(varbinary, @OrderId))

            insert into dbo.EncryptedReport(OrderId, BlockNum, Block)

                values(@OrderId, @blockNum, @block)

            set @idx = @idx + @blockSize

            set @blockNum = @blockNum + 1

        END

    END

END

To decrypt the document, you must:

Open the symmetric key, then
Reassemble the document by
- Decrypting each block, and
- Concatenating the results.

Here's the definition of the stored procedure that will fetch the document associated with a particular OrderId:

CREATE PROCEDURE [dbo].[usp_FetchEncryptedReport] 

    @OrderId int = 0

AS

BEGIN

    -- SET NOCOUNT ON added to prevent extra result sets from

    -- interfering with SELECT statements.

    SET NOCOUNT ON;

    Declare @numBlocks int

    Declare @rowIdx int

    Declare @text varchar(max)

    Declare @block varbinary(8000)

    Open Symmetric Key keyReportEncryption

        decryption by certificate certDocEncryption

    set @rowIdx = 1

    set @text = ''

    set @numBlocks = (select IsNull(max(blocknum), 0) from dbo.EncryptedReport where OrderId = @OrderId)

    BEGIN

     WHILE @rowIdx <= @numBlocks

        BEGIN

            set @block = (Select Block from dbo.EncryptedReport where OrderId = @OrderId and BlockNum = @rowIdx)

            set @text = @text + Convert(varchar(max), DecryptByKey(@block, 1, Convert(varbinary,@OrderId)))

            set @rowIdx = @rowIdx + 1

        END

    END

    Select @text

END

Finishing Your Security Infrastructure

There is no such thing as a perfectly secure architecture that provides a 100% guarantee against the theft or compromise of your sensitive documents. An authorized user can give up his network credentials to a social engineering attack; an unpatched or badly configured database server might allow a hacker to gain administrator access to it; a trusted network administrator can steal data outright. What SQL Server 2005 does very well is protecting data "at rest"--i.e., sensitive documents stored in your database. By following the first 3 steps in this article, you should be able to get a solution up and running quickly. Just remember: make sure you implement other forms of network security (including the last 4 steps mentioned in the introduction) to make your entire infrastructure for handling sensitive documents resilient against theft and hacking.

If you have suggestions or questions, please leave a comment! Thanks for reading; I trust you have learned something useful from this article.

A T-SQL Stumper

Chris Falter — Sat, 15 Dec 2007 23:20:24 GMT

We have a web service that serves as a gateway to a remote service provider. Part of the design is a log of all interactions between our service and the remote service. We keep the log in a SQL Server table with this definition:

CREATE TABLE [dbo].[Request](
    [RequestId] [int] IDENTITY(1,1) NOT NULL,
    [TxCode] [char](2) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
    [ResponseText] [varchar](max) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
    [ResponseTime] [datetime] NULL,
    [CompanyId] [varchar](25) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
CONSTRAINT [PK_Request] PRIMARY KEY CLUSTERED
(
    [RequestId] ASC
) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

The TxCode field identifies what type of transaction is taking place. If the transaction is an AddressServer report, the TxCode will be 'AV'; if the transaction is a Location report, the TxCode will be 'L'; and so forth. The ResponseText field holds the XML document provided by the remote service (if any). The CompanyId field holds a unique identifier within our systems, usually a quote number or a policy number.

The XML in the AddressServer reports identifies the algorithm used for address matching and the algorithm used for "geocoding" (determining the latitude and longitude of an address). Recently I had to determine the frequency of the various AddressServer algorithms reported within a certain time period. My first inclination was to write a query that operated on every AddressServer report within the desired range, like so:

Select count(*) from dbo.[Request]
where requestid between 1000000 and 1100000
and txcode = 'AV'
and (charindex('MatchLevel_StreetInCity', ResponseText) > 0)
and (charindex('GeocodeMatch="Postal"', ResponseText) > 0)

However, I realized that this query did not operate over the correct set of records, because for each unique CompanyId, our gateway might order several AddressServer reports--and the only report we were interested in was the final one ordered. So I needed to restructure the query so it would not operate on all the 'AV' reports, but instead only on the final report for a particular CompanyId.

I was able to come up with a clever query, but before I share it with you, I want to give my readers the opportunity to figure out the correct query for themselves. If you want to show the world you're a SQL guru, leave a comment with a query that operates on the correct set of records. For those who do not have the time or inclination to work on this puzzler, I will share my own query in a comment to this post on Monday afternoon, December 17, 2007.

EDIT: 17 December, 2007 - Since no one wanted to claim SQL guru status by leaving a comment, I will provide my answer in the post, where I can apply helpful formatting.

The fundamental task is this: for each CompanyId you have to find the record (with txCode 'AV') that is most recent. Since the RequestId field is auto-incremented, the most recent record is also the one with the greatest RequestId. Thus we can reduce our set of records by restricting it to those with the Max(RequestId) for any particular CompanyId:

First draft of T-SQL solution:

Select count(*)
from dbo.[Request]
where requestid in
   (select max(requestId) from dbo.[Request]
    where requestid between 1000000 and 1100000
    and txcode = 'AV'
    group by companyId)
and (charindex('MatchLevel_StreetInCity', ResponseText) > 0)
and (charindex('GeocodeMatch="Postal"', ResponseText) > 0)

Notice that once we have sub-selected the correct request IDs based on TxCode and requestId range, we can remove those conditions from the (outer) select's predicate.

However, we still have a problem: what if the last 'AV' transaction for a particular companyId is not within the range? For example, the rquestId for the first 'AV' record for a companyId might be 1099999, and the final one might be 1100001 (which is not between 100000 and 1100000). How do we deal with this problem? It's time to use a clause which is probably used far too little in the world of SQL programming--the "having" clause:

Second draft of T-SQL solution:

Select count(*)
from dbo.[Request]
where requestid in
   (select max(requestId) from dbo.[Request]
    where requestid between 1000000 and 1100000
    and txcode = 'AV'
    group by companyId
    having count(*) > 1)
and (charindex('MatchLevel_StreetInCity', ResponseText) > 0)
and (charindex('GeocodeMatch="Postal"', ResponseText) > 0)

The having clause guarantees that the "group by" clause will exclude any records for a companyId that has only a single 'AV' record within the range.

Of course, if it is possible for a companyId to have 3 or more 'AV' records, 2 of which are within the range and one which falls after the range. Since the last one (outside the range) is the one we would need--and we are not going to be able to include it unless we do not provide an upper bound (not the intention of this query), we have to add one final condition:

Final T-SQL solution:

Select count(*)
from dbo.[Request]
where requestid in
   (select max(requestId) from dbo.[Request]
    where requestid between 1000000 and 1100000
    and txcode = 'AV'
    group by companyId
    having count(*) > 1)
and requestid not in (select requestId from dbo.[Request] where txcode = 'AV' and requestid > 1100000)
and (charindex('MatchLevel_StreetInCity', ResponseText) > 0)
and (charindex('GeocodeMatch="Postal"', ResponseText) > 0)

Now we have excluded any 'AV' records which are not truly the final one for a particular companyId. The task is finished!

But why does it have to be so hard? :) Comments welcome!

Opportunistic SOA (or How to Make Your Cool New Code Survive the Hype Cycle)

Chris Falter — Wed, 11 Apr 2007 01:55:00 GMT

Robin Harris just posted a near-heresy on his ZDNet blog: he thinks that SOA (Service-Oriented Architecture) is overhyped, and will die a painful death as businesses discover the difficulties in monetizing it and making it reliable. Come on now--hype, in our industry? Please!

Actually...I agree that we are indeed well into a hype cycle regarding web services, and we would do well to make sure that SOA doesn't become the next "Golden Hammer." This is a term coined by Brown, Malveau et al. in their terrific book, Anti-Patterns, and it refers to the idea that, to a little boy with a hammer, everything is a nail. And likewise, to an organization that has drunk deeply of the SOA hype, every new software initiative can be an opportunity to create a web service.

Yet certain types of problems are best solved with web services, provided that you recognize the opportunities and take advantage of them. Implementing services as the right opportunities arise is what I call "Opportunistic SOA."

Whenever one of our lines of business asks for some new functionality, we ask ourselves, "Will any other lines of business want this functionality as well?" If the answer is "yes" or "most likely," we typically invest the 25% extra effort needed to implement the functionality as a service. Then when the next LOB asks for the functionality, we get to reap the reward of a vastly simpler and faster implementation. And when a third LOB wants it, we're laughing our way to the bank.

Obviously, there are some situations where only a web service will do. If you want to access some functionality across the internet, you're going to use a web service, period, end of story. Also, if systems on disparate platforms require the same functionality, you will probably write a web service, because the alternative is to write the same functionality multiple times in multiple languages for multiple platforms.

When systems on the same platform on the corporate network need the same functionality, though, there are tried-and-true approaches that compete with web services. In this situation, how do you distinguish a good SOA opportunity from an opportunity to make your users' lives harder while playing with a new technology?

* A good web service hides the complexity of interacting with another system. This is why our shop does not use web services to make database calls, even though it involves inter-process communication. A database has so many possible interfaces (stored procs, table structures, etc.) that inserting an additional web service layer between the client and the server usually does not simplify anything. The only time a web service interface might make sense for a data layer is when the data has to be transported over HTTP due to network considerations. If you're inside the firewall, in other words, it's hard to justify writing a web service data layer.

* A good web service simplifies the management of inter-process interactions by allowing all updates to occur in a single place--namely, the web service. Remember, you can simplify an interface to a complex system with the facade design pattern, which relies on a facade class to handle the complexity of interacting with some other system. If it does not make sense to manage the interaction via a single system at a single point in your network, you should not write a web service; you should write a facade class library instead and compile it into your codebase.

Our shop has used web services to provide interfaces to services from outside vendors. For example, some of our homeowners insurance lines acquire outside data regarding fire protection class, building code effectiveness grading scale, and distance to shore before underwriting a particular home. The vendor's XML interface is quite complex, so we use an internal web service to simplify the interface. Using a web service (instead of a facade class) also allows us to manage all the vendor updates and the configuration elements in a single place.

Another situation that was ripe for a web service was our need to convert PCL-formatted documents from our legacy system into PDF format for a web application. We got a server license for a conversion component, wrapped it in a web service interface, and were able to perform the necessary conversion from any system on any platform. And recently, when we discovered that the vendor component was not as reliable as we had hoped, we rejoiced that we did not have to re-write and re-compile code in a dozen different places to use a new vendor's (server-licensed) component. We have only one system whose source needs to change--the web service. We are planning to change the web service implementation in the near future, and all the client systems will get the benefit of the improved reliability without blinking an eyelash.

As is often the case with a new architecture, there is a sweet spot between the extremes of SOA hype (on the one hand) and total rejection of change (on the other). I hope this article will help you find it.