Performance & Tuning

Health Monitoring in ASP.NET

Chris Falter — Sat, 07 Jun 2008 01:03:56 GMT

While I was busy customizing Microsoft's Exception Management Application Block to classify and log all exceptions thrown in our web and Windows apps, and writing instrumentation code that published timing events via System.Diagnostics.Trace, Microsoft was busy writing ASP.NET Health Monitoring. Microsoft has more resources, so their product is a little more advanced and customizable. Here's what it provides:

An event model: There are event classes for web request failures, for authentication failures, for errors, for SQL errors, for heartbeats, and so forth. You can subclass the WebBaseEvent in order to define your own custom events, as well.
A provider model: Health monitoring can publish events with 4 built-in providers (email notification, SQL Server, WMI, and event log). The WMI provider is especially interesting, but you will have to write your own application to manage and report WMI events, according to Microsoft.
A customization model: You can edit web.config in order to map events to providers and to configure providers (for example, to specify a connection string for the SQL Server provider).

Since we have already customized so much infrastructure, especially by writing a bug portal that makes use of the customized Exception Management block, we will probably not be migrating to the ASP.NET Health Monitor any time soon. However, for those of you who have been whistling in the dark with respect to the health of your web applications and services, get busy! I highly recommend Scott Allen's post as a starting place for your efforts.

A Tool for Troubleshooting Web Apps and Web Services

Chris Falter — Mon, 28 Jan 2008 19:20:45 GMT

Recently I discovered a tool that I had needed for a long time. Fiddler, a freeware product created and supported by Microsoftie Eric Lawson, can be used to debug HTTP traffic from any web browser (or client application). Here's how you can use it:

1. View/Analyze HTTP traffic.

One of our web apps creates a new window and performs a javascript-driven auto-logon to a vendor application. When we reported an issue to the vendor, they started asking questions about whether we were using the correct version of their javascript module. Rather than attempting to debug the javascript, I simply used Fiddler to capture all the HTTP traffic between my browser and the vendor's web server. Fiddler formatted the data quite nicely, which made verifying my browser's behavior quite simple. Fiddler was also able to save the relevant portions of the HTTP conversation to a log file, which I forwarded to the vendor. The vendor was able to determine quickly that the error was in the behavior of their web server, rather than in any incorrect use of javascript on our part.

Just one day later, another vendor questioned whether one of our web services was correctly returning error data. Source code-level debugging in the web service would have done me no good, since I had to verify the data being transmitted on the wire. But Fiddler made the investigation easy. I used Fiddler to capture an interaction between a test client and the service, and quickly verified that our service was returning the correct error data.

You can also use Fiddler to view all the details of a secure HTTP interaction. Try doing that with netmon! Just configure Fiddler to view HTTPS traffic, and it will hook any secure HTTP transactions. It uses the "man-in-the-middle" strategy: it sets up a secure HTTP transaction between the browser and itself, then transmits the data it receives to the target URL over secure HTTP. Sitting in the middle, it is able to view all the data in the browser-server conversation. You just have to tell your browser to ignore the any certificate errors associated with the temporary certificate that Fiddler generates.

2. Generate test scripts.

The Visual Studio 2005 Test Edition has a tool that allow you to generate a test script for a web app by capturing the interaction between a browser and a web app. However, this tool only captures interactions (such as form submits) generated by clicking on links and buttons. So how do you capture javascript-driven AJAX interactions? Use Fiddler!

3. Analyze performance enhancements.

If you are trying to figure out how various browser settings will affect a website's performance, Fiddler can help. You can script Fiddler to intercept browser interactions and alter metadata, which allows you to perform a "what if" analysis on the fly. And Fiddler has a built-in timer to measure wall clock time during the interaction.

The Fiddler site has extensive documentation, including some good QuickStart videos. If you've been looking for a good tool to analyze HTTP traffic, look no more; go directly to the Fiddler download page and get started!

How To: Instrument Your Code to Time Its Use of Resources

Chris Falter — Fri, 13 Jul 2007 19:46:46 GMT

Last week we went through a "fire drill" when users of one of our web applications started receiving "Gateway Timeout" errors (HTTP Error 504). This error indicates that a gateway "server" (any host that forwards the HTTP request, including a router or web server) did not receive a timely response from a server that is supposed to respond. Naturally we didn't have a clue as to which host in the connection chain from users' browsers to our web farm was reporting the error, but the leading candidate for the ultimate culprit was our web application. Hence the fire drill; I had to do whatever it took to find out what might be causing a slow response. Well, there went my 4th of July celebration!

A quick Performance Monitor analysis showed that the web servers and web application were not causing any delays. Average CPU utilization was less than 6%, and the web application itself does not have any shared internal resources that could serve as a bottleneck. So if there was a bottleneck, it had to be in the interface between our web application and some external resource.

That in turn meant that I had to insert instrumentation code to log the time spent in those interfaces. In the .NET Framework 2.0, you accomplish this with the System.Diagnostics.Stopwatch class:

Stopwatch timerCall = null;

string rawResult = String.Empty;

try
{
    VendorProxy p = new VendorProxy();
    if (useTimers)
    {
        timerCall = new Stopwatch();
        timerCall.Start();
    }
    rawResult = p.GetVendorData(reqText);
}
finally
{
    if (useTimers)
    {
        timerCall.Stop();
        Trace.WriteLine(
            DateTime.Now.ToString() + "," +
            responseType + "," +
            seibelsId + "," +
            timerCall.ElapsedMilliseconds.ToString()
            );
        Trace.Flush();
    }
}

In the class' static constructor, we set the value of the useTimers boolean according to a BooleanSwitch in the web.config :

private static bool useTimers;

static VendorGatewayService()

{

    BooleanSwitch timerLoggingSwitch = new BooleanSwitch("TimerLogging", "Determines whether to log timing info");

    if (timerLoggingSwitch.Enabled)

        useTimers = true;

}

See the MSDN documentation for an example of how to configure a BooleanSwitch in a config file. Likewise, you may configure a TraceListener in a config file as well. We chose to write our output to a comma-separated value (CSV) file due to its ease of use with Excel. By logging the responseType (i.e., name of method) and our customer's ID, we would be able to determine which methods might be too slow, and which of our customers were affected. (After all, they sometimes call to ask questions!)

If your application still uses .NET Framework 1.0 or 1.1, you should use Peter Bromberg's HiPerfTimer class in place of the Stopwatch class. Do not use the built-in TimeSpan class, whose precision is very poor. Also, notwithstanding the MSDN documentation, it is impossible to configure a TraceListener in a config file prior to .NET Framework 2.0. As a result, in my 1.1 web app I added the following key/value pairs in the appSettings element:

<add key="UseTimers" value="true" />

<add key="TraceLogName" value="c:\\projects\\dev\\agentweb\\mysoln\\1.4\\spweb\\Timer.log" />

Here's how I used them in the Application_Start handler of global.asax.cs:

protected void Application_Start(Object sender, EventArgs e)

{

    bool useTimers = false;

    string strUseTimers = ConfigurationSettings.AppSettings["UseTimers"];

    if (strUseTimers != null)

        useTimers = Boolean.Parse(strUseTimers);

    if (useTimers)

    {

        string traceLogName = ConfigurationSettings.AppSettings["TraceLogName"];

        if (traceLogName != null && traceLogName != String.Empty)

        {

            StreamWriter file = File.AppendText(traceLogName);

            TextWriterTraceListener log = new TextWriterTraceListener(file);

            Trace.Listeners.Add(log);

        }

    }

}

Now that my colleague/manager Chris Glasser has identified the main bottleneck, I can edit the config files to turn off the logging. And if another resource contention issue rears its nasty-looking head, I can edit the config files to turn the logging back on. Very convenient. (For you Michael Jackson fans, it's as "easy as A-B-C.")

Want to know what the root problem was? It wouldn't be fair for me to leave my readers hanging, would it? Okay, you've stuck with me through this article; I'll reward your faithful readership.... The insurance agents' requests were never even reaching our web application due to networking issues at one of big carriers. When I returned from my delayed vacation, you could almost see the steam come out of my ears when I learned this. Have any of you ever worked like crazy to solve a problem, only to discover that the root cause was something outside your control? Leave a comment so we can commiserate!

Solved: The Mystery of DataView's Poor Performance with Large Recordsets

Chris Falter — Wed, 16 Aug 2006 06:24:00 GMT

Well, maybe I solved it. At least I have a reasonable hypothesis.

Reader “Ghassan” postulated in a comment to my earlier post that it's the extra burden of creating an array of DataRowView that puts the DataView approach at a disadvantage to DataTable.Select. However, a glance at DataView's default indexer in Lutz Roeder's wonderful Reflector utility shows that the array of DataRowView is not created while the constructor is executing. Instead, it is created when and if the DataView indexer is called. The DataView indexer calls its private GetElement method in order to return a DataRowView:

private DataRowView GetElement(int index)
{
      if ((index < 0) || (index >= this.recordCount))
      {
            throw ExceptionBuilder.GetElementIndex(index);
      }
      return this.RowViewCache[index];
}

And the RowViewCache uses lazy instantiation for the array of DataRowView:

internal DataRowView[] RowViewCache
{
      get
      {
            if (this.rowViewCache == null)
            {
                  this.rowViewCache = new DataRowView[this.Count];
                  for (int num1 = 0; num1 < this.Count; num1++)
                  {
                        this.rowViewCache[num1] = new DataRowView(this, num1);
                  }
            }
            return this.rowViewCache;
      }
}

In other words, unless and until your code attempts to get a DataRowView via the indexer, the DataView will not create the array of DataRowView at all. And my code did not call the indexer until after the 9 seconds had elapsed. So we must eliminate this hypothesis. Of course, Ghassan, I did not supply a code listing, you could not have known this. Believe me, I appreciate reader comments, and I wish I got more of them! Please keep posting comments. Please.

So what are the DataView constructor and RowFilter setter doing while the clock ticks off 9 seconds? Here's what the default constructor does:

It initializes default settings. This involves maybe a couple of dozen assignment statements, which can't take more than a few milliseconds.
It instantiates an internal DataViewListener, which creates delegates for the DataTable events that need to be monitored. Again, this should take only a few milliseconds at most.
It instantiates an internal Index, which is an array of int members that index into the DataTable. This involves:
- Instantiating the array, which should (once again) involve modest computing resources.
- Setting the bounds of the array by applying any filter criteria. For the default constructor, there is no filter, so it's a one-step operation; the size is equal to the DataTable's record count.
- Sorting the array.

Then my next line of code, the call to the DataView.RowFilter setter, will create a second internal Index. When index sets the bounds of its array, it will have to apply the RowFilter criteria. This should be comparable to the computing resources used by the DataTable.Select method.

The only significant difference in computing resources, then, must come from the fact that instantiating a DataView and setting its RowFilter involves sorting a rather large array of record indices twice. Let's take a closer look at that sort:

private void Sort(int left, int right)
{
      int num1;
      do
      {
            num1 = left;
            int num2 = right;
            int num3 = this.records[(num1 + num2) >> 1];
            do
            {
                  while (this.CompareRecords(this.records[num1], num3) < 0)
                  {
                        num1++;
                  }
                  while (this.CompareRecords(this.records[num2], num3) > 0)
                  {
                        num2--;
                  }
                  if (num1 <= num2)
                  {
                        int num4 = this.records[num1];
                        this.records[num1] = this.records[num2];
                        this.records[num2] = num4;
                        num1++;
                        num2--;
                  }
            }
            while (num1 <= num2);
            if (left < num2)
            {
                  this.Sort(left, num2);
            }
            left = num1;
      }
      while (num1 < right);
}

We see that the Sort method implements the Quicksort algorithm. It chooses an arbitrary midpoint, places all the values lower than the midpoint's to the left and all the higher values to the right. Then it applies itself recursively to the left and right sections. Eventually it will recurse down to sections that cannot be divided into smaller sections (i.e., when a section consists of a single array member), at which point the sort has completed. Using big-O notation, we can say that this algorithm executes in O(n log n) time, which is about as efficient as you can expect from a sorting algorithm. As long as each iteration of the sort divides the set of indices into two equal parts, we are dealing with a logarithm of base 2. So for 122,000 records (approximately 2 raised to the 17th power), the algorithm will call the Index.CompareRecords method approximately 122,000 x 17 times, which is a little over 2 million times. And each call to Index.CompareRecords traverses a call stack several layers deep as it wends its way through the object model. Do this whole scenario twice and you can chew up 9 seconds.

At least I think so. To prove all this, you would have to instrument the System.Data code and check run-time performance with a testing tool. So if there's a Computer Science student out there who wants an interesting research project, have some fun with Mono and tell us what you find. Anyone?

Why You Might Prefer a DataView to an Array of DataRecord

Chris Falter — Tue, 08 Aug 2006 07:47:00 GMT

Before we get too obsessed about the performance advantages of an array of DataRecord over a DataView, let's remember that a DataView behaves differently than an array of DataRecord. No matter how you modify the data in an array of DataRecord, the array bounds will always be the same. However, modifying a field in a DataRecord may cause it to disappear from a DataView, if the modification causes the record to be filtered by the DataView.Filter property. Or if you modify a DataRecord in a DataTable, you might cause it to suddenly appear in a related DataView.

If you need this kind of dynamic, event-driven update of set membership, you are going to have to use a DataView, regardless of the performance considerations. If you don't need the dynamic set membership update, then performance considerations would suggest you use an array of DataRecord.

Array of DataRecord vs. DataView: A Dramatic Difference in Performance

Chris Falter — Sat, 05 Aug 2006 06:34:00 GMT

I spent much of last weekend running tests against a table with 122,000 records. The folks at Citibank and Geico would just yawn at that amount of data, but where I work that's fairly heavy lifting. As I was tuning and validating the tests, I used a bit-field flag ("TestCompleted") to track whether a record had already been tested. No need to do work twice, eh?

Because this was a one-time set of tests, I used a TableAdapter (the simplest possible code) to grab all the records, then I needed to filter out all the records that had already been processed. My initial solution was to use a DataView; just set the Filter property with a filter expression ("TestCompleted = false"), and voila--instant filtration! Although it wasn't quite instant....my computer required 9 seconds to perform the operation.

For reasons I won't go into here, I had to switch to using an array of DataRecord returned by the DataTable.Select method. The filter expression ("TestCompleted = false") was simply passed as a parameter to the Select method, rather than to the DataView.Filter setter. So I ran the test again, and....whoa!

I had my array of DataRecord in just one second. That's a whole order of magnitude faster than the DataView approach!

Frankly, I don't know why the DataView.Filter approach is so much more computationally expensive than the DataTable.Select approach. And perhaps for small amounts of data, the two approaches would be equally good--or the DataView might even be better and faster. I didn't test the small recordset scenario, so I don't know that either. (Note: I have subsequently posted an analysis of the reason why the DataView is so much slower for those who are interested.)

What I do know is this: if you're filtering a large amount of data in a DataTable, and you care about system performance, you will want to filter your data by using DataTable.Select (to get an array of DataRecord), rather than using DataView.Filter.