Chris Falter

Health Monitoring in ASP.NET

Chris Falter — Sat, 07 Jun 2008 01:03:56 GMT

While I was busy customizing Microsoft's Exception Management Application Block to classify and log all exceptions thrown in our web and Windows apps, and writing instrumentation code that published timing events via System.Diagnostics.Trace, Microsoft was busy writing ASP.NET Health Monitoring. Microsoft has more resources, so their product is a little more advanced and customizable. Here's what it provides:

An event model: There are event classes for web request failures, for authentication failures, for errors, for SQL errors, for heartbeats, and so forth. You can subclass the WebBaseEvent in order to define your own custom events, as well.
A provider model: Health monitoring can publish events with 4 built-in providers (email notification, SQL Server, WMI, and event log). The WMI provider is especially interesting, but you will have to write your own application to manage and report WMI events, according to Microsoft.
A customization model: You can edit web.config in order to map events to providers and to configure providers (for example, to specify a connection string for the SQL Server provider).

Since we have already customized so much infrastructure, especially by writing a bug portal that makes use of the customized Exception Management block, we will probably not be migrating to the ASP.NET Health Monitor any time soon. However, for those of you who have been whistling in the dark with respect to the health of your web applications and services, get busy! I highly recommend Scott Allen's post as a starting place for your efforts.

How To Create a Windows Form Singleton

Chris Falter — Fri, 06 Jun 2008 21:31:25 GMT

Recently a friend asked me how you might create a Windows Forms application that only allows a single instance per computer. A print driver might make use of this functionality, for example, to launch a print job management dialog whenever a document prints. Never having needed this sort of functionality before, my initial answer wasn't very helpful. But being both curious and disinclined to back down from a technical challenge, I just had to figure this one out.

As I was looking for an inter-process synchronization mechanism, I came across the Semaphore and Mutex in the System.Threading namespace. A semaphore can be used to manage a pooled resource (memory buffer, thread pool, connection pool, etc.) by tracking the number of available resources. You instantiate a semaphore with an invariant maximum corresponding to the quantity of pooled resource entities (e.g., the number of database connections). Whenever a thread wants to use the resource, it calls .WaitOne() on the appropriate semaphore and blocks until the semaphore count is greater than zero. If the count is positive when the call is made, the thread will not need to block at all.

When a thread enters a semaphore, the semaphore decrements its count by one. The calling thread is responsible for calling the semaphore's .Release() method when it has completed using the resource. This allows the semaphore to increment its count--and as a result, gives another thread access to the pooled resource. You must call Release() inside of a finally block as soon as feasible in your codepath; my testing shows that the CLR will not release an unreleased semaphore when the semaphore reference goes out of scope, or even when the thread of execution terminates.

What we need, then, is a semaphore that has a maximum count of one (since we want only one instance of our application to be running) and will increment its count no matter how the thread that entered it terminates. Happily, this would be a pretty good one-sentence description of the Mutex class. The thread that calls WaitOne on a mutex ("mutual exclusion") gains ownership when it enters the mutex. It can release the mutex at any time by calling the ReleaseMutex method, but should it fail to do so (whether by logic error or run-time failure) before it terminates, the CLR will release the mutex on its behalf. As a result, the mutex should prove more reliable for our purposes.

Here is the singleton code in a nutshell:

using System;

using System.Windows.Forms;

using System.Threading;

namespace MutexTest

{

    static class Program

    {

        [STAThread]

        static void Main()

        {

            Application.EnableVisualStyles();

            Application.SetCompatibleTextRenderingDefault(false);

            // make sure no other instances of this app are running

            bool mutexIsAvailable = false;

            Mutex m = null;

            try

            {

                m = new Mutex(true, "MutexTest.Singleton");

                mutexIsAvailable = m.WaitOne(1, false); // wait only 1 ms

            }

            catch (AbandonedMutexException)

            {

                // don't worry about the abandonment; 

                // the mutex only guards app instantiation

                mutexIsAvailable = true;

            }

            if (mutexIsAvailable)

            {

                try

                {

                    Application.Run(new Form1());

                }

                finally

                {

                    m.ReleaseMutex();

                }

            }

        }

    }

}

One of three things can happen when the Main() thread calls m.WaitOne(1, false):

If the mutex is unowned (or becomes unowned within one millisecond), mutexIsAvailable is true and the thread enters/owns the mutex.
If the mutex remains owned by another instance of the Windows Form app for the one millisecond duration, mutexIsAvailable is false.
If the mutex is available because another instance abandoned it without releasing it, the method throws an AbandonedMutexException. In this case, the thread owns the mutex and may safely launch the application instance, so it sets mutexIsAvailable true.

If the Main thread owns the mutex, it runs the application, using a try/finally block in order to guarantee a call to ReleaseMutex. While strictly speaking it is not necessary to call ReleaseMutex (the CLR will do so on the thread's behalf when it terminates), I have written the code this way in order to maintain a best practice. In some other situation, the thread that enters the mutex might continue performing more work (or enter a suspended state) after finishing its use of the resource and might not call ReleaseMutex, so it's a good idea to write the code in this fashion.

What do you think? Is this code helpful? Do you have any suggestions for improvement? Leave a comment!

.NET Entity Framework: Is It Ready for Web Prime-Time?

Chris Falter — Tue, 27 May 2008 21:50:29 GMT

Our shop has been looking for a way to simplify the interaction between our domain layer and data layer. The data layer uses typed datasets for all its operations. When it retrieves data, it populates a dataset; when it stores data, it checks the DataRowState of all rows in the dataset, and calls the appropriate stored procedure to insert, update, or delete the data. The domain layer then uses dataset elements as the in-memory backing store for properties and collections. A collection typically uses a DataTable, for example, and a collection member would use a DataRow in the DataTable.

This arrangement has some advantages:

Between requests, domain objects can cache their state in session by caching a dataset.
DataRowState can inform the logic about which CRUD operation to perform on a DataRow. Unmodified data can also be skipped over, yielding improved efficiency.
We have considerable freedom in designing domain classes. We can make a property read-only, for example, if it corresponds to an immutable database field, such as an identity column.

This arrangement also has a big disadvantage, however. We end up with lots of properties that look like this:

public string Phone

{

    get { return dr.PHONE; }

    set { dr.PHONE = value; }

}

This property's one and only use is to map a domain attribute (an entity's phone number) to a particular field in a DataRow (dr). Our domain layer has hundreds of these pass-through properties, which are a very verbose and not especially readable way of mapping between the domain and data layers. So we've been looking for a better way to handle this mapping.

Along comes the .NET Entity Framework (EF), which provides a tool that creates a domain-to-data map in XML format. Our first reaction was: this is just what the doctor ordered! And supported by Microsoft, to boot.

On closer examination, though, the EF toolset has some growing up to do before we could consider using it in our web applications. It can generate only public get/set accessors in the entity domain model (EDM) classes, for example. You cannot make a property read-only, or internal/protected/private, or virtual. You cannot map a non-generated property to a database field. And programming in a straitjacket was not what we had in mind for a new approach.

Second, EDM classes generated by EF do not have any supported way to cache their state. A Microsoftie is working on a tool ("Perseus") that can serialize the state of an entity using an EntityBag<T> instance, but it seems somewhat incomplete (and definitely not supported) at the moment. Since our web apps do cache domain entity state, adopting the EF right now would entail an important risk.

What alternatives exist? We can license DevForce EF, which is built on top of the EF and seems to offer pretty much everything that it lacks. Or we can try NHibernate, an open-source .NET port of the Hibernate tool which is quite popular in the Java development world.

Or maybe we can just wait for the EF to mature. Microsoft has big plans for the EF, and intends to make REST-oriented web services, Reporting Services, workflows, etc. EDM-aware. So an EF investment today can yield important dividends in the future.

Even though I cannot render a definitive answer, I hope that this discussion will help readers make an informed discussion about their ORM choices in the .NET world. What do you think? Leave a comment with your insights or questions!

BOOK REVIEW: Don't Make Me Think, 2d Ed. by Steve Krug

Chris Falter — Wed, 14 May 2008 03:31:39 GMT

You've mastered web forms and controls. You've prototyped a Silverlight 2.0 application. AJAX? You're all over it. But have you really learned how to design a good web page or web site?

Steve Krug's "Common Sense Approach to Web Usability" provides surprising and sometimes counterintuitive principles that every good website must follow. Krug preaches the importance of removing clutter in order to make the purpose and functionality of a site (or page) clear--and happily, he practices what he preaches in this remarkably lucid book. Here are some of Krug's key insights:

Don't make users think! Your job is to make sure users do not have to puzzle over a site's purpose, or how to use it. Krug offers the search functionality at Amazon.com as a great example of this principle in action. A user does not have to decide what type of search she wants (by author, by title, by ISBN, etc.); instead, she can just enter whatever text interests her, and Amazon offers a list of matches, ranked by relevance.
Users don't behave the way you think they do. You've been poring over your site--reading everything ten times or more--so you tend to think that users will do the same. But they don't. Instead, the users...
- "don't read pages, they scan them." (In fact you only decided to read this review after you scanned the intro and decided it would be worth your while.)
- "don't figure out how things work, they muddle through."
Design pages for scanning. Since users are going to treat your site like a billboard going by at 70mph (rather than a textbook that they carefully work through), you should learn to design great billboards!
- Create a clear visual hierarchy. Highlight the important stuff, and indicate relationship by grouping.
- Use conventions. If your site design conforms to what users generally expect, they can more easily understand it at a glance.
- Break pages into clearly defined areas.
- Make what's clickable obvious. Use arrows or underlines to indicate that text is clickable, for example.
- Minimize noise. Prefer clarity to pizzazz.
Give users simple\mindless choices. It's okay to make a user traverse 4 or 5 links to get to his desired destination as long as each choice along the way is clear.
"Omit needless words." "Get rid of half the words on each page, then get rid of half of what's left" is Krug's Third Law of Usability. More words just make the site look more daunting, which can discourage the user in a hurry. Krug's Third Law has 2 corollaries:
- Happy talk must die. "We're so glad you're at our site! We think you'll like your experience here..." Oops, you just lost your user, who's too busy to keep reading.
- Instructions must die. Users almost never read them anyway (remember, they don't figure out, they muddle through). So keep instructions brief and simple.

Krug then discusses some details about 2 features every site must have: navigation and a home page. Krug recommends that the sections and subsections of a site be indicated with a clear set of links or tabs on the home page, and that the navigation remain available no matter where the user goes. (This answers the questions: what can I do on this site? and Where can I go from here?) Each page should have a visible name, and the site should a breadcrumb trail with arrows between the levels in order to allow a user to know where he's at in the site's hierarchy. (These answer the questions: where am I at now? and How do I get back to where I was before?).

Krug's discussion of the home page acknowledges that the forces conspiring against simplicity are enormous in any ambitious site, because there are so many organizational interests competing for the valuable home page real estate. Krug provides some useful tips for managing the problems, though, by suggesting how to use logos, taglines, and home page navigation.

Next Krug addresses the often religious arguments that site development teams often endure (pulldowns or menus? Flash animation or simple text?) with this simple advice: forget the arguments and start testing! Simple usability testing will reveal the key problems with a site, and often they have nothing to do with what the development team has been arguing about. Krug believes that frequent tests are more important than comprehensive (often expensive tests), and he offers advice on how to do testing on a tight budget (use 3 or 4 subjects; use an inexpensive screen recorder like Camtasia; try to get all the project stakeholders to observe). He concludes with an extremely useful script of an interaction between a test subject and a test guide.

Those who have already read the first edition will be pleased to know that Krug has included some very helpful new material in the second edition. Is your site accessible to sight-impaired users? If not, Krug offers a top 5 list of tips for making sites accessible, along with a pointer on how to use Cascading Style Sheets to make your site more accessible. Not to mention that CSS makes your site a lot easier to manage.... And what do you do when your boss (or the customer) wants you to do something that violates every known law of web usability? Just compose an email that borrows liberally from one of Krug's friendly "here's how it should be done" missives!

Krug sprinkles his book with examples of sites that work well (and a few that don't) to illustrate his ideas. He often offers a few "How does (or does not) this page implement the principles we've been discussing?" tests, with his own answers on the following pages. I found these examples to be enormously helpful. And Krug offers a wonderful set of additional resources for those who want to pursue the subject further, both within the text and in a notated bibliography at the end.

Because I am a long-time web user and web developer, I thought I understood just about everything I needed to know about usability to design a good site. Then I read this book, and learned a ton. If you work on web sites in any way (whether as designer, developer, or tester), the few hours you spend on Steve Krug's little gem will pay rich dividends.

How To: Encrypt and Manage Documents with SQL Server 2005

Chris Falter — Fri, 09 May 2008 04:13:56 GMT

When one of our departments decided to store sensitive reports, the architecture I first considered was storing them, unencrypted, in our imaging system. The imaging system cannot manage encrypted documents, but its security capabilities allow it to to deny unauthorized users access to the reports. This ability to restrict access seemed to meet our security requirements.

Then I started asking questions about data backups. Offsite backups can be a major security vulnerability, as we have recently learned. The names, addresses, social insurance numbers, and (in many cases) bank account details of almost half of England ended up who-knows-where last year because database backups disappeared in the postal system. And our company, like any responsible organization, backs up our data and stores it offsite. So leaving the sensitive data unencrypted was not acceptable.

Fortunately, SQL Server 2005 comes with native encryption capabilities that are simple to use. In the absence of an affordable imaging system with built-in encryption capabilities, here's the approach I hammered out:

Create a table in a SQL Server 2005 database to store the encrypted documents.
Create a symmetric key in the same database. To facilitate recovery from database failures, store the instructions for re-creating the key in a safe, remote location.
Create stored procedures that use the key to encrypt/decrypt documents that are stored in the table.
Create a service account under which the service that stores the report will run.
Create a (presumably internal) website for displaying the reports.
- Use the same service account for the site's app pool identity.
- Restrict access to authorized groups via Windows authentication.
- Require secure http so that a network trace cannot intercept decrypted documents on their way to the browser.
On SQL Server, deny permissions for the stored procedures and table to anyone other than the service account. The security context of the backup job must also be granted the db_backupoperator role on the database, of course.
Implement network encryption between SQL Server and any services that insert or select sensitive documents, using IPSec or SQL Server Network Encryption.

Below is a diagram of the system that allows authorized users to view sensitive documents.

There is scant MSDN documentation for accomplishing the first 3 steps, but the rest are well understood in the world of Microsoft development. The remainder of this article will therefore focus on creating and managing the tables, keys, and stored procedures.

Step #1: Create a Table to Store the Encrypted Documents

Several considerations drive the design of the table:

The column that stores a block of encrypted data must have the datatype of varbinary(8000) because the result of an encryption operation is always a varbinary(8000).
Unless you are dealing with very small documents, 8000 bytes will not hold an entire document. Therefore a single document will require multiple records, which will be associated with one another by a common unique identifier. A foreign key to the business entity associated with the document would be a good candidate. In our system, the encrypted document is associated with an "Order," so the identifier is an OrderId.
In order to thwart brute force decryption attacks, you should "salt" the encryption with a unique integer for each report. Unless your sensitive documents are very large, the foreign key (OrderId) will serve the purpose nicely.

Here's the table definition that emerges from these considerations:

CREATE TABLE [dbo].[EncryptedReport](
   [OrderId] [int] NOT NULL,
   [BlockNum] [int] NOT NULL,
   [Block] [varbinary](8000) NULL,
CONSTRAINT [PK_EncryptedReport] PRIMARY KEY CLUSTERED
(
   [OrderId] ASC,
   [BlockNum] ASC
)
) ON [PRIMARY]

Step #2: Create a Symmetric Key

To encrypt data in SQL Server 2005, you may use a certificate or a symmetric key. A certificate provides somewhat stronger protection than a symmetric key, so it is the encryption method of choice for a small amount of data (such as a Social Security Number). However, its far greater demand for server resources makes it impractical for large amounts of text or other data. Therefore, if you are encrypting a sensitive document, you should create a symmetric key and encrypt it with a certificate. At run-time, SQL Server will use the certificate to decrypt the key, then use the key to encrypt or decrypt the document.

Before SQL Server can encrypt or decrypt a sensitive document for you, it must traverse a chain of encrypted secrets:

A certificate available only to the SQL Server process identity encrypts SQL Server's Service Master Key.
The Service Master Key encrypts the Database Master Key.
The Database Master key encrypts a certificate ("certDocEncryption" in this example).
The certificate encrypts a symmetric key ("keyReportEncryption" in this example).
(Finally!) the symmetric key encrypts your sensitive documents.

SQL Server stores all of these secrets for you, except for the process identity's certificate (which is stored in the Windows certificate store). Consequently, you do not need to worry about making a "secret" available to your run-time environment by, for example, storing it in a configuration file. Database backups will not include the process identity's certificate (which the operating system maintains for you), so the only way the bad guy that steals your backup can decrypt your sensitive documents is by a brute force attack.

This design does place certain responsibilities on your IT organization:

You must be very careful about using this design with a virtual server. While creating an image of an entire virtual server (Windows OS, SQL Server, and data) facilitates quick disaster recovery, it can also facilitate the theft of sensitive documents, since the virtual server image will have everything (from Windows certificate down to the symmetric key) that a hacker needs to read the data.
To recover data in the event of a server failure, you must be able to re-create the symmetric key. Wyile it is impossible to back up a symmetric key, you can always create an identical key on any instance of SQL Server 2005 by issuing the "CREATE SYMMETRIC KEY" statement with the same algorithm, key source, and identity value. Consequently, you must store the statement used to create your symmetric key in a secure location, such as a safe deposit box at a local bank. And although this statement is quite obvious, I'll say it anyway: don't store the statement with your data backups!

Step #3: Create Stored Procedures

Encrypting a sensitive document is a four-step process:

Open the symmetric key.
Break the document into blocks that, when encrypted, will consume less than 8000 bytes. When I applied Microsoft's published formula to my encryption key process, I concluded that 7876 bytes was the maximum size of unencrypted data that I could safely fit into a varbinary(8000).
Encrypt each block.
Insert each encrypted block into the table.

Here's the definition of the stored procedure that does the trick:

CREATE PROCEDURE [dbo].[usp_StoreReportWithEncryption] 

    @OrderId int, 

    @ReportText varchar(max) = ''

AS

BEGIN

    declare @idx int

    declare @textLength int

    declare @blockSize int

    declare @blockNum int

    declare @block varbinary(8000)

    declare @keyGuid uniqueidentifier

    Open Symmetric Key keyReportEncryption

        decryption by certificate certDocEncryption

    set @keyGuid = Key_GUID('keyReportEncryption')

    set @blockSize = 7876

    set @blockNum = 1

    set @idx = 1

    set @textLength = datalength(@ReportText)

    BEGIN

        WHILE @idx < @textLength

        BEGIN

            set @block = EncryptByKey(@keyGuid, SUBSTRING(@ReportText, @idx, @blockSize), 1, Convert(varbinary, @OrderId))

            insert into dbo.EncryptedReport(OrderId, BlockNum, Block)

                values(@OrderId, @blockNum, @block)

            set @idx = @idx + @blockSize

            set @blockNum = @blockNum + 1

        END

    END

END

To decrypt the document, you must:

Open the symmetric key, then
Reassemble the document by
- Decrypting each block, and
- Concatenating the results.

Here's the definition of the stored procedure that will fetch the document associated with a particular OrderId:

CREATE PROCEDURE [dbo].[usp_FetchEncryptedReport] 

    @OrderId int = 0

AS

BEGIN

    -- SET NOCOUNT ON added to prevent extra result sets from

    -- interfering with SELECT statements.

    SET NOCOUNT ON;

    Declare @numBlocks int

    Declare @rowIdx int

    Declare @text varchar(max)

    Declare @block varbinary(8000)

    Open Symmetric Key keyReportEncryption

        decryption by certificate certDocEncryption

    set @rowIdx = 1

    set @text = ''

    set @numBlocks = (select IsNull(max(blocknum), 0) from dbo.EncryptedReport where OrderId = @OrderId)

    BEGIN

     WHILE @rowIdx <= @numBlocks

        BEGIN

            set @block = (Select Block from dbo.EncryptedReport where OrderId = @OrderId and BlockNum = @rowIdx)

            set @text = @text + Convert(varchar(max), DecryptByKey(@block, 1, Convert(varbinary,@OrderId)))

            set @rowIdx = @rowIdx + 1

        END

    END

    Select @text

END

Finishing Your Security Infrastructure

There is no such thing as a perfectly secure architecture that provides a 100% guarantee against the theft or compromise of your sensitive documents. An authorized user can give up his network credentials to a social engineering attack; an unpatched or badly configured database server might allow a hacker to gain administrator access to it; a trusted network administrator can steal data outright. What SQL Server 2005 does very well is protecting data "at rest"--i.e., sensitive documents stored in your database. By following the first 3 steps in this article, you should be able to get a solution up and running quickly. Just remember: make sure you implement other forms of network security (including the last 4 steps mentioned in the introduction) to make your entire infrastructure for handling sensitive documents resilient against theft and hacking.

If you have suggestions or questions, please leave a comment! Thanks for reading; I trust you have learned something useful from this article.

Want to Write Great Business Software? Create a Great Domain Model!

Chris Falter — Tue, 25 Mar 2008 21:01:28 GMT

When you are coding in a hurry, it is very tempting to write business logic in the first place that comes to mind, such as a button click handler. However, for all but the simplest systems, such a practice leads very quickly to a chaotic system whose business logic is scattered like the ash from an erupting volcano. So let's take a step back and see how you can develop an application more intelligently.

"Divide and Conquer"

“Divide and Conquer” is a useful metaphor for organizing the effort of developing a complex system. A software architect can assign major system responsibilities to various components or layers (which I will refer to as a “part”). As long as the responsibilities of a part are understood, a developer can develop the part in such a way as to satisfy those responsibilities. And as long as the parts integrate well, a useful system can emerge.

“Divide and Conquer” has many advantages:

A team can work in a coordinated fashion on several parts of the system simultaneously, without stepping on each other’s toes.
Assigning a limited set of related responsibilities to a system part simplifies its design.
Modifying a system is easier when the part that is responsible for the functionality that must be changed can be identified.
Ultimately, the complexity of the system decreases as its various responsibilities, and the strategies for handling them, become clear. Unnecessary complexity is what can make “the last 10%” of a project go into deep cost and time overruns, as quality issues and poor design make their effects known.

“Divide and Conquer” is the driving force behind creating logical layers of a user interface, an application layer, a data layer, and a domain model in an enterprise software application.

While the advantages of separating UI, application, and data logic into their own layers are well understood and frequently practiced, the point behind creating a domain model is not as widely grasped. A domain model defines the business entities represented by a system, the activities associated with them, the relationships between them, and the rules and constraints that govern them. These are clearly very important aspects of the system, even though (from the end user’s perspective) they are not as visible as the user interface, application flow, or database. I will illustrate the importance of a domain model by examining how insurance endorsements work in the homeowners point-of-sale system that my team has developed.

Example of a Complex Business Domain

Let’s start by examining the data model for endorsements:

An endorsement can be associated with either an application (future policy) or with a risk (such as a dwelling or a vehicle). Therefore the Endorsement table has 2 foreign keys: one that relates an endorsement to an application (via the Quotenumber) and the other that relates it to a risk (via the RiskId).
Endorsements come in a bewildering variety. Some endorsements have no related data; some have one or more limits; some have one or more attributes, few of which are common between endorsements. Clearly it is unwise to attempt to create a single table with identified columns for each of the possible limits or attributes that an endorsement may possess. In addition, any attempt to create a unique table for each endorsement’s associated attributes is going to result in a very large number of tables—along with the responsibility of defining a new table each time a new endorsement type is created. Therefore, the design we chose was to create 2 dependent tables for Endorsement: an EndorsementLimit table and an EndorsementDescription table. Each table has 3 fields: an EndorsementId (foreign key back to the associated endorsement), a “type” field that allows the system to distinguish between all the related descriptions (or limits) of an endorsement, and the associated data field (a numerical limit or a varchar description). Note that segmentation and history are irrelevant to a point-of-sale system, so the schema is not designed with that capability.
Some endorsements are concerned with parties to an insurance application other than the applicant, such as an additional interest or an additional insured. Since these additional entities reside in a different table (the OtherParty table), we use a join table (EndorsementParty) to manage the association between Endorsement and OtherParty.

This portion of the database schema is represented graphically below:

So if we have a data model, why do we need a domain model? Why can’t some UI code simply write its data directly to the tables? Clearly this one way of designing a system—you can, after all, do anything with software (provided you have sufficient time and resources).

Let us consider, though, the knowledge about the business domain that the point-of-sale application must manage with regard to endorsements:

The relationship of endorsement attributes to an endorsement. A homeowners “HO 05 28” (golf cart) endorsement has several related attributes: a limit (cart value), a make, a model, a serial number, and an indicator of whether collision coverage is included. Each one of these attributes is stored in its own record in the EndorsementLimit or EndorsementDesc table. The DescType field distinguishes the four EndorsementDesc records from one another; however, the system must somehow keep track of which DescType is used for which attribute.
Differences between the format of stored data and how it is used. In a database system that does not have a built-in boolean type (such as the DB2 flavor used at my shop), there are probably as many different ways to store boolean data as there are programmers. True vs. false could be represented as “0” vs. “1”, or as “1” vs. “2”, or as “T” vs. “F”, or as “t” vs. “f”, or as “true” vs. “false”, or as “Y” vs. “N”, etc. (In one of my consulting engagements, I actually saw all of these plus a couple more in a single application.) To handle a golf cart endorsement’s indicator of collision coverage, the system must be able to reliably map between the stored data and a run-time choice of true/false.
The relationships between endorsements and other entities. Some endorsements are related to the risk being insured (such as the earthquake endorsement), and some are related to the (potential) policy itself (such as an “Additional Insured” endorsement).
The relationships between endorsement attributes. The “HO 04 65” endorsement (scheduled property) contains a set of schedules; each schedule is designated by a letter such as “A” or “J” and has a description (the personal property being insured) and a limit. Since descriptions and limits are stored in separate tables, the system must have some way of identifying and associating the schedules’ descriptions and limits.
Rules and constraints on various endorsements. The 6 schedules on a “HO 04 65” endorsement have base (default) limits, for example, such that the total limit of a schedule can be described as the base limit plus an additional limit. Mathematically speaking, we use the formula Total Limit = Base (fixed by rule) + Additional (stored in EndorsementLimit table). So the system must be able both to provide base, additional, and total limits (given the stored additional limit) and to update the stored additional limit (based on user entry).

Complexity Resolved: A Domain Model

The beauty of our team's domain model is that it completely hides all of this complexity from the user interface.

An endorsement class is defined for each type of endorsement, and it provides an interface that is radically simple for the UI developer to use. To give just one example, the golf cart endorsement provides a boolean CollisionIncluded property that can be mapped to a checkbox.
All of the other issues (which EndorsementDesc record the indicator is stored in, how the boolean is mapped to a varchar, etc.) are managed inside an endorsement class—frequently by using capabilities inherited from the EndorsementBase class or borrowed from utility classes in the domain layer.
Responsibilities are clearly grouped and assigned in this way of designing a system. Tasks and attributes that are common to all endorsements are the responsibility of the EndorsementBase class. The tasks and attributes that are unique to a particular endorsement are the responsibility of the corresponding subclass of EndorsementBase (which already has the common endorsement capabilities via its inheritance relationship with EndorsementBase). Neither of these classes needs to know anything about the application or user interface which will use them, which in turn need to know nothing at all about the internal implementation details of an endorsement class. The diagram below depicts the assignment of some endorsement responsibilities to classes in our domain layer.
The association of an endorsement with a policy or with a risk is managed via its membership in an EndorsementCollection that is accessed via the HomeApplication object or the HomeApplication.Home object (of type HomeRisk), respectively. The relationship between these various entities is depicted below.

Conclusion

There are some things that should never leak. Space shuttle O-rings. Tires. Diapers. And now that you've read this article, add your business domain model to the list. Always isolate your business domain logic in a cohesive model; do not let the logic leak into a "smart UI" or into the data layer! If you follow this rule, you will save yourself and your team much time and effort when you need to modify the business logic in your application.

Also, keep the importance of your domain model in mind when it is time to choose a development tool. A rapid application development (RAD) tool that can only design screens and logic flows might be a very good choice for building a very simple system that needs little complexity in its business logic. However, it is very difficult to envision how a RAD tool would be able to represent the relationships and constraints inherent to a complex business domain, such as homeowners insurance endorsements. Given the inherent limitations of a RAD tool, the complexity that my team's domain model currently manages quite capably would have to spill out into the flows and screens themselves.

While the result may initially seem counterintuitive, a RAD tool that at first glance makes everything look simple might very well in the end produce a system far more complex than a system produced with a set of tools that offer a robust and complete set of programming capabilities. And this unanticipated complexity can make the systems you develop far more difficult to implement on time and to extend with new customer-requested capabilities.

Refactoring to Comply with the DRY principle (Don't Repeat Yourself)

Chris Falter — Fri, 07 Mar 2008 19:45:40 GMT

If you're not living DRY, you're not a good programmer.

Okay, I'm not talking about alcohol consumption! I'm talking about the Don't Repeat Yourself principle--a principle every programmer should live by. Recently I did a code review for a project we outsourced, and demonstrated how you can refactor bad cut-and-paste code into well-designed code. You will note that the capabilities of the MagicStringTranslator class really help to reduce code clutter. Without further ado, I present the relevant portion of my code review:

...your branching logic has 12 lines of code in each branch except the final one. I am inserting an excerpt for reference:

if (m_record.Text.Substring(13, 6).Trim() == "IDENT")

{

writer.WriteBreak();

writer.WriteFullBeginTag("Table style='width: 100%' border='1'");

writer.WriteFullBeginTag("tr style='background-color:#ccffcc;'");

writer.WriteFullBeginTag("td style='width: 100%;' colspan='2' align='center'");

writer.WriteFullBeginTag("span style='font-size: 9pt;");

writer.WriteFullBeginTag("strong");

writer.Write("SUBJECT INFORMATION");

writer.WriteEndTag("strong");

writer.WriteEndTag("span");

writer.WriteEndTag("TD");

writer.WriteEndTag("TR");

writer.WriteEndTag("Table");

}

if (m_record.Text.Substring(13, 6).Trim() == "SUMRY")

{

writer.WriteBreak();

writer.WriteFullBeginTag("Table style='width: 100%' border='1'");

writer.WriteFullBeginTag("tr style='background-color:#ccffcc;'");

writer.WriteFullBeginTag("td style='width: 100%;' colspan='2' align='center'");

writer.WriteFullBeginTag("span style='font-size: 9pt;");

writer.WriteFullBeginTag("strong");

writer.Write("SUMMARY");

writer.WriteEndTag("strong");

writer.WriteEndTag("span");

writer.WriteEndTag("TD");

writer.WriteEndTag("TR");

writer.WriteEndTag("Table");

}

if (m_record.Text.Substring(13, 6).Trim() == "COLLT")

{

// this code and several more near-identical branches are omitted for brevity’s sake

}

Eleven of these lines of code are identical. This is bad for 3 reasons:

What would happen if one of the lines of code had a bug? You would have to fix it in each branch, and be sure to do it the same way in each one. This code structure is bug-prone.
The structure implies that each branch does different things, when the code in fact does essentially the same thing. As a result, the code structure is misleading.
The code ends up being way too verbose; it is saying the same thing over and over and over. As a result, the code structure is hard to read and understand.

In fact, your code does suffer from an insidious little bug. You are comparing a substring of length 6 to a literal of length 5.

if (m_record.Text.Substring(13, 6).Trim() == "IDENT")

This only happens to work when the 6th character is a blank, because in this circumstance the Trim() method reduces the string to length 5, and the comparison is valid. What would happen if the 6th character did not happen to be a blank?

But don't try to fix this bug in the seven places where it occurs! There are better ways to improve the code. One way is to restrict the branching logic to the one line of code, then use that one line together with the other 11 exactly once.

string title = String.Empty;

switch (m_record.Identifier)

{

case "IDENT":

title = "SUBJECT INFORMATION";

break;

case "SUMRY":

title = "SUMMARY";

break;

case "COLLT":

title = "COLLECTION ITEMS";

break;

case "ACCNT":

title = "TRADE ACCOUNT ACTIVITY";

break;

case "MSSGE":

title = "MESSAGE";

break;

case "EMPLY":

title = "EMPLOYMENT";

break;

case "INQHS":

title = "INQUIRY HISTORY";

break;

}

writer.WriteFullBeginTag("Table style='width: 100%' border='1'");

writer.WriteFullBeginTag("tr style='background-color:#ccffcc;'");

writer.WriteFullBeginTag("td style='width: 100%;' colspan='2' align='center'");

writer.WriteFullBeginTag("span style='font-size: 9pt;");

writer.WriteFullBeginTag("strong");

writer.Write(title);

writer.WriteEndTag("strong");

writer.WriteEndTag("span");

writer.WriteEndTag("TD");

writer.WriteEndTag("TR");

writer.WriteEndTag("Table");

An even better way is to delegate the relationship between magic string (e.g., “INQHS”) and description (e.g., “INQUIRY HISTORY”) to a class specialized for this purpose. Then you can simply use the class directly, without writing any branching logic at all. In this situation, define a MagicStringTranslator subclass for the Identifier property:

public class IdentifierCode : MagicStringTranslator

{

public const string SUBJECT_INFO = "IDENT";

public const string SUMMARY = "SUMMARY";

public const string COLLECTION_ITEMS = "COLLT";

public const string TRADE_ACCOUNT_ACTIVITY = "ACCNT";

public const string MESSAGE = "MSSGE";

public const string EMPLOYMENT = "EMPLY";

public const string INQUIRY_HISTORY = "INQHS";

}

Then make the class’ Identifier property of type IdentifierCode:

public IdentifierCode Identifier

{

get{ return new IdentifierCode(segments[(int)Fields.Identifier].Data); }

set { } // no-op setter provided so that XmlSerializer can serialize this property

}

Finally, use the Identifier directly, without any need for branching logic:

writer.WriteFullBeginTag("Table style='width: 100%' border='1'");

writer.WriteFullBeginTag("tr style='background-color:#ccffcc;'");

writer.WriteFullBeginTag("td style='width: 100%;' colspan='2' align='center'");

writer.WriteFullBeginTag("span style='font-size: 9pt;");

writer.WriteFullBeginTag("strong");

writer.Write(m_record.Identifier.Description);

writer.WriteEndTag("strong");

writer.WriteEndTag("span");

writer.WriteEndTag("TD");

writer.WriteEndTag("TR");

writer.WriteEndTag("Table");

And you're done.

A Good Solution for "Magic String" Data

Chris Falter — Thu, 06 Mar 2008 22:08:45 GMT

Dealing with vendor data (or your own) in the form of "codes" can pose significant challenges. You must ensure that your source code remains readable, that data are properly validated, and that data can be displayed as user-friendly descriptions. The built-in solutions (named constants and enums) help, but they have some significant shortcomings. If you derive a class of named constants from the MagicStringTranslator class, though, you can vanquish all 3 challenges in one fell swoop!

Egghead Cafe has kindly published the improved version of my article here. Enjoy!

Make Magic Strings Easy to Understand and Type Safe

Chris Falter — Wed, 27 Feb 2008 13:40:16 GMT

As organizations pass data back and forth, they often use codes to represent the data. For example, a marital status of divorced might be represented as "D", married as "M", and so forth. You have to solve three problems when you are dealing with magic strings:

When you write logic to handle the data, things can get chaotic on a hurry if you are not careful; the use of literal magic strings in your source code can make it incomprehensible.
You can get into trouble by passing a string that is not in the set of valid codes as a parameter to a method that is expecting one of the codes. Since the parameter type is typically string, the compiler will not help you detect the error.
You often have to translate codes into it description so a user of your system will know which data have been gathered.

Using named constants can help make your source code more readable, but you still have more work to do: you must write extra logic to check whether a string belongs to the set of valid codes, and to translate a code into a description that a user can understand. If you declare your named constants in a class that inherits from the MagicStringTranslator class, though, you will get the data validation and the translation for free. Check out the article I wrote for the details!

"New" Statement Considered Harmful

Chris Falter — Sat, 16 Feb 2008 02:59:48 GMT

All of us have probably written code like this:

Foo f = new Foo();

And what could be simpler? As long as the logic in the constructor is simple (or better yet, the constructor is empty), it would seem that the simplest code is the best, so just use the constructor. Certainly the MSDN documentation is rife with code that uses public constructors. You can probably find plenty of public constructors used right here on my blog. Why invest the effort in writing (and using) a factory class that will probably never do anything useful, other than call a public constructor?

In his excellent podcast entitled "Emergent Design: The Evolutionary Nature of Software Development," Scott Bain of Net Objectives nevertheless makes a strong case against the routine use of public constructors. The problem, notes Scott, is that the use of a public constructor ties the calling code to the implementation of Foo as a concrete class. But suppose that you later discover that there need to be many subtypes of Foo, and Foo should therefore be an abstract class instead of a concrete class--what then? You've got a big problem, that's what; a lot of client code that has been making use of Foo's public constructor suddenly becomes invalid.

Exposing a public constructor, observes Bain, allows code to violate the open/closed principle. This principle states that software should be open for extension, but closed for modification. Restating this in more mundane terms, developers should design their code in such a way that extending it (for example, making a class more useful by defining it as abstract and subclassing it as appropriate) does not provoke side effects elsewhere--and that's the "closed for modification" part in a nutshell.

Bain points out that there is a simple, low-cost practice that avoids the public constructor's violation of the open/closed principle. Just give a class constructor protected scope, and define a static Create method that uses the constructor--that's it. Here's class Foo's new skeletal definition in C#:

public class Foo
{
protected Foo() {}

public static Foo Create() { return new Foo(); }
}

How hard was that? At the cost of a single line of code, you get benefits well beyond the already mentioned ability to transform the class into an abstract base class:

If object construction becomes more complex, you can use a factory class inside the Create method.
You can implement the Dependency Injection pattern, which facilitates unit testing with mock objects.
You can use Microsoft's Policy Injection Application Block to implement cross-cutting concerns such as logging, authorization, and validation.

Unfortunately, you may occasionally encounter a corner case where you must define a public constructor. The .NET Framework's XmlSerializer cannot be used on a class with no public default constructor, since XmlSerializer must instantiate the class in order to discover the public properties that must be serialized. If you plan to serialize a class using XmlSerializer, then, you must make the constructor public--but you should mark it with the ObsoleteAttribute to warn fellow developers not to use it:

public class Foo
{
[Obsolete("Do not use--provided only for use by XmlSerializer. Client code should call Foo.Create().")]
public Foo() {}

public static Foo Create() { return new Foo(); }
}

If due to habit a fellow programmer calls the public constructor (against your intentions), the C# compiler will emit a helpful warning message. Of course, there are programmers who ignore warnings, but that is the subject of a different post.

Agile Software Development: A "Value-Up" Approach

Chris Falter — Thu, 14 Feb 2008 03:22:43 GMT

I am reading Software Engineering with Microsoft Visual Studio Team System by Sam Guckenheimer, the group product planner for Microsoft's Visual Studio Team System. I am finding the book very helpful and thought-provoking, so I want to share my thoughts as I work through the book. This essay contains my thoughts on the first chapter.

The standard models of software project management--in particular, the "waterfall" methodology at the root of the Software Development Lifecycle (SDLC)--are based on the management approaches of other engineering disciplines, such as civil engineering. Consider the project management forces in play when you are building a bridge:

Well-Understood Design - You can build a new bridge that is virtually identical with an existing bridge. The design elements of a bridge, the milestones the project will cross, and the features that must be included (move a certain volume of traffic safely) contain no surprises.
Low Variance in Estimates - Hundreds of similar bridges have been built previously, so it is not difficult to estimate cost and effort with great accuracy.
Little Ability to Provide Incremental Value - A half-finished bridge is of no use to anyone!

Guckenheimer refers to SDLC methodology as a "work-down" approach. After you have completed your design, you draw up the task list, estimate the effort, and draft a work plan. Managing the effort (especially the critical path) is based on this up-front work plan--hence the use of the term "work-down."

The problem with the work-down approach is that most software projects face quite different forces than the typical bridge building project:

Design issues are not often solved prior to building the system. The new system is not identical to any other system--at least any system whose project plan and resource expenditures are known. If an almost identical system were available, the customer would probably buy it off the shelf.
System requirements are incomplete, at best. As users start to interact with the system and discover its capabilities, they will generate new ideas for features. Thus the development team will face a demand for shifting requirements. Of course, if the requirements change, you can throw the original estimation of cost and effort out the window. On the other hand, if the project team resists shifting requirements due to its up-front investment in an extensive design phase and work estimation (you have heard of "scope creep," right?), the team is at risk of delivering a system with lots of features that offer little value to the customer.
The up-front estimation of time and effort may be very imprecise due to poorly understood design challenges and customer requirements.

The Agile development movement is a response to these difficulties in the traditional work-down approach. The Agile approach is a "value-up" approach; it seeks to provide increasing value, incrementally, as a system is being built. Essentially, an agile approach targets (in collaboration with the customer) a specific set of valued features in a project iteration. This iteration should not have a long duration; in the Scrum methodology, for example, the team's iteration is a one-month "sprint." This short duration minimizes the risks involved with a traditional software project, since the customer is (presumably) obtaining the highest value features first, and the design/coding tasks being estimated are an easier-to-understand portion of an ever-evolving system.

Guckenheimer provides a table of differences between work-down and value-up approaches that I summarize here:

*Work-Down vs. Value-Up Methodologies*
	Work-Down (Traditional) Approach	Value-Up (Agile) Approach
Planning and Change Management	Invest heavily in up-front planning to get it right, then monitor the project against the plan. Be vigilant to guard against change.	Change is inevitable, so embrace it. Do just enough planning to understand risks and manage the next project iteration.
Progress Measurement	Task completion	Customer-approved deliverables. Be skeptical of interim mileposts that do not deliver value to the customer.
Definition of Quality	Conformance to requirements. To reduce risk, invest heavily in getting the right requirements up-front.	Value to the customer. Since the customer may not understand what value a system can deliver before it is built, deliver incrementally and do not seek comprehensive requirements up-front.
Intermediate Work Products	Models, requirements documents, etc., help decompose the effort into tasks that can be estimated. Therefore their completion provides useful milestones in measuring project progress.	You want just enough models and documents to maximize the flow of work towards completing deliverables that a customer values. Intermediate work products have no other value and are not counted as deliverables.
Evaluation of Project Team	Compare individuals' work output against the project plan. Individuals that are struggling (like Wally in the Dilbert cartoons) are motivated to obfuscate and shift blame.	Team members are motivated when the value of their work is measured by their team's deliverables. Transparency (all team members can see the overall team's performance) is highly valued.

Visual Studio Team System (VSTS) facilitates Agile project management by providing a work item list that is visible to all team members. The status of any item is tracked by the status of the latest tests (such as FIT scenarios) linked to requirements, and by the system build version(s) against which a test suite is run. Often project plans are hard to manage because updates and tracking (like source code check-ins) are manual operations; VSTS eases the burden by providing automated updates to work items via check-in changesets and automated build verification tests (BVTs).

VSTS work item tracking can also answer questions about quality concerns and project run rates. For example:

Project Management Question	VSTS Feature That Provides an Answer
Are there any components that are poorly designed?	Check the bug count for the various assemblies.
Are there any components that are inadequately tested?	Check the code churn metric and the code coverage percentage in the automated tests
Can we deliver the projected functionality on time?	Don't rely on code check-in metrics only; instead, make sure that the bug backlog for desired features is decreasing. If the open bug count is not decreasing over time (or worse, is increasing), you're likely not going to deliver on plan, even if the team believes the project is 95% finished.

Our organization has not been fully aware of how VSTS can facilitate agile software development; maybe it's time to start finding out.