Karthik (from our Mobilizer Madness group) found me at the computers, and dragged me away from blogging this morning's sessions to go eat lunch. After lunch, I went to:
CSI448 Optimizing Scalabilty, Performance and Availability with Systems Built on the .NET Framework
This was a great talk by Ingo Rammer, because as he said this was based on current technology rather than talking about how Microsoft Scalability Server 2008 is going to solve all of our problems. He works as an architecture trouble-shooting consultant, so this job is to go in and look at problems after deployment. Here are Ingo's techniques as presented:
Don't Assume Anything (Even if you coded it/designed it/whatever)
1. Trace the Network Interactions
Use packet sniffing or network monitoring tools: Ethereal, TcpTrace, Fiddler, or ProxyTrace. The “wire never lies“. You may think you're making one call, but on the network it blows out to 500 calls. He showed us a real-life example of a simple 200 row fetch that transferred 400 packets. After simple optimization it went down to 40 packets! The culprit in this case was marking a remoting type as both Serializable and deriving from MarshallByRefObject...doing both required the framework to hit the remote server for every property invocation. You could see it in the ethereal trace--it was nasty.
2. Trace your SQL
Hook up SQL Profiler to your DB server, and see what's going on. In the example Ingo showed, a client app was using a DataSet to make some updates, then shipping it back to the DB. Well, the server had to update the database from the DataSet, and it took ~1000 SQL statements to do it. With properly crafted SQL statements (instead of using dataset) it went down to ~10 statements. Using foreach or iteration to update a DataSet is bad...use SQL or storedproc instead.
This method is also useful for finding SQL commands that are in inner loops of server-side code. Heh, I know this from personal experience.
3. Memory Profiling
Use CLR Profiler 2.0 to look into the allocations per method call in your program. Use the Allocation Graph to check this. Watching memory allocation is very important in ASP.NET apps, where all users are sharing the memory on a machine. Ingo mentioned that on 32-bit machines, each ASP.NET process has realistic access to only 1.5 GB of ram. When you realize this is shared across all users of the application, you can imagine the impacts of a memory intensive app. (I'm going to have to hunt down the reference for this...not sure why the 1.5GB limit)
4. In Proc Profiling with DevPartner Studio, RedGate ANTS, or AQ Time (mmm, $$$). No demos on this point.
Ingo suggested to try using load balancing during development so as to be prepared for clustering during deployment. Check into Window's Network Load Balancing (NLB)...it's built into the OS and super easy to setup. No other hardware is required other than your cluster members. There is a short article describing NLB.
Finally, avoid “HttpIsm's” Sometimes we try to make HTTP do stuff it wasn't intended to do. Some points that can trip you up:
- Authentication methods: If you are doing basic authentication, your client may have to make each request twice (once to get the 401 error, then again to send the request with the credentials). In .NET, turn on PreAuthenticate = true on the web service proxy.
- Have IIS serve a smaller 401 error page instead of the nicely formatted user-friendly page that most automated clients aren't going to care about.
- If you are using a web service server behind a web server, you gotta do something in web.config OTHERWISE that maching will only be able to handle 2 requests at a time. I missed the key to config, so I'll hunt it down and post it later.