My company recently purchased a software called dynaTrace to be used for performance diagnosis. It primarily started as a Java tool but they do support .Net as well. Since my company is entirely .Net based I will explain the product from that point of view.
The application works by using MSIL injection to put essentially a header and footer into each method you have instrumented in an application. If you do not know what MSIL Injection is here is a great overview on the PostSharp website, and here is a more detailed but not as accessible explanation of how to do it.
So as your applicaiton is running each instrumented method reports back to a diagnostic server which collects all the method calls and then strings them together. There is a client application that connects up to the server and lets you see what is going on in a particular execution. Each seperate transaction flow in your application, what dynaTrace calls a purepath, is knitted together so you can follow it all the way through.
The product can follow across web service, MSMQ, and remoting calls so that everything is seen together as one transaction. The current version cannot do WCF but their next release (hopefully 1st quarter 2009) will.
So, let's say you had an instrumented ASP.Net website that when some user makes a request, it calls a web service which is also instrumented, which then contacts the database. With dynaTrace we would see that as one request, it would be relatively transparent when the code executed across to the web service. The calls the web service makes to the database are seen because ADO.Net is instrumented, but the database server itself is not directly instrumented.
So now I can see the total time of the web request , all the pieces of it, and what database calls are made. With that information you can figure out what the big contributers are very easily.