Posts
208
Comments
1144
Trackbacks
51
WinDbg real world example

I recently had to debug a problem for my current client which was exceptionally weird and I was able to utilize WinDbg to help get to the bottom of the problem relatively quickly.  Basically the application in question is an asp.net application that takes a custom object and puts in in an MSMQ message (which currently uses the default binary serializer).  The class in question is marked with the Serializable attribute, has a couple of primitive members (e.g., ints, strings) and a couple of NameValueCollections.  One of the NameValueCollections is assigned directly from the HttpRequest.Headers NameValueCollection like this:

myObject.Headers = request.Headers;

One day we started getting this error:

System.Runtime.Serialization.SerializationException: Type 'System.Web.HttpHeaderCollection' in Assembly 'System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' is not marked as serializable.

This made absolutely no sense since, like my class, the NameValueCollection class was also marked with the Serializable attribute.  Not only that, but where was the System.Web.HttpHeaderCollection coming from?

So since I was already debugging, I took a memory dump of the asp.net process from the command line like this:

cscript.exe adplus.vbs -hang -p 1808 -quiet -o c:\temp

The "1808" was simply the PID of my w3wp.exe process and I output the dump to the c:\temp directory.

I then launched WinDbg and opened the crash dump that was produced in the previous step. First, you have to run (the sos extensions is what allows WinDbg to understand .NET 2.0):

.loadby sos mscorwks

So the first thing I wanted to see was all the types in memory and their corresponding method tables so I ran:

!dumpheap -stat

A snippet of the results:

00000642802ae6c0        5          320 MyNamespace.Foo

(just using Foo here for the class name.  The real output included the fully qualified namespace with correct object name).  The 00000642802ae6c0 is the method table for my Foo object. The 5 shows that we currently have 5 of my Foo objects loaded into memory.  The 320 is the total size (320/5 tells me each of my objects is taking up 64 bytes).

Now I can narrow in on my issues a little bit. This shows me all 5 of the memory addresses for my Foo objects:

0:000> !dumpheap -mt 00000642802ae6c0
------------------------------
Heap 0
         Address               MT     Size
total 0 objects
------------------------------
Heap 1
         Address               MT     Size
00000000c0012e18 00000642802ae6c0       64    
total 1 objects
------------------------------
Heap 2
         Address               MT     Size
000000010053b830 00000642802ae6c0       64    
total 1 objects
------------------------------
Heap 3
         Address               MT     Size
0000000140235c60 00000642802ae6c0       64    
0000000140430fd8 00000642802ae6c0       64    
00000001405275d0 00000642802ae6c0       64    
total 3 objects
------------------------------
total 5 objects
Statistics:
              MT    Count    TotalSize Class Name
00000642802ae6c0        5          320 MyNamespace.Foo
Total 5 objects

So now I'm going to pick one of these five objects and drill down further.  I'll pick the one highlighted in blue that resides at 00000001405275d0:

0:000> !do 00000001405275d0
Name: MyNamespace.Foo
MethodTable: 00000642802ae6c0
EEClass: 00000642802baf08
Size: 64(0x40) bytes
 (C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\root\e22c2559\92c7e946\assembly\dl3\fed41db4\000e24c3_21cac701\MyAssemblyName.DLL)
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
0000000000000000  4000028        8                       0 instance 0000000140527610 requests
0000064274e85620  4000029       10 ...meValueCollection  0 instance 0000000140528d60 headers
0000064274e85620  400002a       18 ...meValueCollection  0 instance 0000000140527838 extraRequestData
0000000000000000  400002b       20                       0 instance 0000000140527a38 userStateData
000006427881edc8  400002c       28        System.String  0 instance 0000000000000000 endOfLineData
0000064278825318  400002d       30       System.Boolean  0 instance                0 isOptOut

So I know the "headers" member is the one I want for focus on so I inspect that next:

0:000> !do 0000000140528d60
Name: System.Web.HttpHeaderCollection
MethodTable: 00000642bcefad78
EEClass: 00000642bcf3be38
Size: 120(0x78) bytes
 (C:\WINDOWS\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
0000064278825318  4001165       44       System.Boolean  0 instance                1 _readOnly
00000642788426f0  4001166        8 ...ections.ArrayList  0 instance 0000000140528f78 _entriesArray
000006427881f650  4001167       10 ...IEqualityComparer  0 instance 00000000ffffc148 _keyComparer
00000642788448c8  4001168       18 ...ections.Hashtable  0 instance 0000000140529000 _entriesTable
0000064274e82360  4001169       20 ...e+NameObjectEntry  0 instance 0000000000000000 _nullKeyEntry
0000064274e9e7e8  400116a       28 ...se+KeysCollection  0 instance 00000001405481c0 _keys
0000064278857210  400116b       30 ...SerializationInfo  0 instance 0000000000000000 _serializationInfo
000006427882b338  400116c       40         System.Int32  0 instance               10 _version
000006427881d280  400116d       38        System.Object  0 instance 000000014054ad48 _syncRoot
000006427881f748  400116e      a70 ...em.StringComparer  0   shared           static defaultComparer
                                 >> Domain:Value  0000000000136900:NotInit  0000000000177700:00000000ffffc110 <<
00000642788d8ba8  4001179       48      System.Object[]  0 instance 0000000000000000 _all
00000642788d8ba8  400117a       50      System.Object[]  0 instance 0000000140548160 _allKeys
00000642bceef310  400101b       58 ...m.Web.HttpRequest  0 instance 00000001405265f0 _request
00000642bceef7b8  400101c       60 ....Web.HttpResponse  0 instance 0000000000000000 _response
00000642bcf3ad30  400101d       68 ...IIS7WorkerRequest  0 instance 0000000000000000 _wr

And NOW we see something extremely interesting.  The headers member was supposed to be a NameValueCollection so why is my !do showing it to be an HttpHeaderCollection and what is this HttpHeaderCollection anyway and how is this System.Web DLL even compiling? So at this point, it's time for us to crack open trusty old Reflector and find its class definition:

internal class HttpHeaderCollection : HttpValueCollection 

So this is an internal class to the System.Web DLL but it's being returned for a NameValueCollection property.  Also notice that it is NOT marked with the [Serializable] attribute (which was our problem to begin with).  It inherits from the HttpValueCollection (which is yet another internal class) and the HttpValueCollection (while it DOES have the Serializable attribute) inherits from the NameValueCollection object and THAT is how the code is compiling because of course this is a perfectly legal use of polymorphism.

So...in the end, all of these classes are marked with the Serializable attribute except for the HttpHeadersCollection (which I believe to have been an oversight on Microsoft's part) which is causing our serialization problems and we cannot control the internal code of System.Web.  The fix ends up being pretty trivial - just iterate the NameValueCollection and adds the string primitives so that all serialization is happy.

But the real moral of the story is that this is a good real world example that shows how to debug an issue that, at first glance, seems impossible. Using WinDbg like this is a very repeatable process that can help you get to the bottom of memory issues (and others) quite fast.

posted on Tuesday, August 14, 2007 9:16 PM Print
Comments
No comments posted yet.

Post Comment

Title *
Name *
Email
Comment *  
Verification

View Steve Michelotti's profile on LinkedIn

profile for Steve Michelotti at Stack Overflow, Q&A for professional and enthusiast programmers




Google My Blog

Tag Cloud