Tray on the Web

How To Determine Which Application Pool W3WP.EXE Belongs To on an IIS 7 Web Server

Tray Harrison — Thu, 05 Nov 2009 19:23:40 GMT

This is a follow up to my earlier post that details how to track down which applications pools your w3wp.exe processes belong to on IIS 6 servers. The command is new on Windows 2008 Server, so that calls for a new blog post.

So imagine you log into your web server, pull up task manager and you see multiple processes running named w3wp.exe. You know this is a web server so you guess these processes may have something to do with the websites on the server, but you don’t have any idea which sites they belong to. Being able to identify them is especially important if you are troubleshooting an issue and one of these guys is taking up all of your resources.

First you’ll need to open task manager and make sure the PID column is showing. That way you can match up the processes with their application pools. To do this go View—>Select Columns and check the box for PID.

Here is where things get different on 2008 server. Open a command prompt and navigate to C:\Windows\System32\Inetsrv. From there, run the command appcmd list wp. You’ll get an output of all running worker processes along with the names of the application pools that they are running in.

If you’re trying to track down an issue, match the PID from the output of this command up to what you see in task manager. From there you’ll be on your way to getting things resolved.

How To Determine Which Application Pool W3WP.EXE Belongs To on an IIS 6 Web Server

Tray Harrison — Thu, 05 Nov 2009 18:58:48 GMT

Today I was troubleshooting an issue in which one of our production SharePoint web applications wouldn’t load and displayed a very un-helpful “Unexpected Error” when you hit the site from a browser. I logged into the server, pulled up task manager, and could instantly see that something wasn’t right.

The CPU usage was holding steady in the 90 – 100% range, and nearly all of it was being chewed up by a single worker process (w3wp.exe). Unfortunately task manager isn’t smart enough to tell you which website that worker process belongs to, so you have to do a little further investigation before you can begin to troubleshoot. There are a couple of ways to do this, but I’ll outline what I feel is by far the easiest.

The first thing you’ll need to do is make sure that Task Manager is showing you the PID (process identifier) for everything running. If it’s not, then all you need to do is go to View—>Select Columns and put a check mark in the PID box. So for example’s sake, lets imagine that the w3wp.exe running under the PID 12012 is consuming all of your server’s CPU. Thankfully this isn’t the case currently on my server.

Now that we know what the offending process is, we need to figure out which application pool it belongs to. To do this, open a command prompt and navigate to c:windowssystem32. From there you can run the command IISAPP.VBS. For example:

As you can see from this point it’s very straightforward to match up the the offending process with the correct application pool. You can then try recycling the application pool or if necessary taking further steps debug your application.

How To Cluster IIS 6 Web Servers

Tray Harrison — Thu, 05 Nov 2009 15:47:41 GMT

First of all, clustering IIS is something you should rarely consider doing. In almost all cases, Microsoft NLB is a much better solution when you need to provide high availability for your web applications. There are situations though where you have limited hardware or infrastructure that is serving multiple purposes and installing IIS on a cluster is your only option.

If you do find yourself in this predicament and have to configure IIS on a cluster, the good news is that the process is actually fairly straightforward. I’m going to leave out all of the MSCS specifics and just assume that you already have the cluster created along with the disk group and virtual IP resources. If you need help with that part of it, technet should be able to point you in the right direction.

Here’s what you do to configure IIS on an active/passive MSCS cluster:

Install IIS on each node of your cluster
Create a folder on the clustered data drive to hold the website content.
If you are going to create a new site to use other than Default Web, verify that the Default Website is either stopped or bound to a local IP on Node A. Then create a website on Node A and bind it to All Unassigned. Point the home directory of your website to the folder you created above.
Create or place a test page in the home directory and verify it comes up
Fail cluster resources over to Node B and create the website over there, pointing to the same home directory and IP as all unassigned
Fail back over to Node A. In cluster Admin right click the cluster group name and select New Resource. Name it IIS Resource and Generic Script for the resource type
Verify both nodes are possible owners.
Select the data disk, IP Address, and Name as dependencies.
Point to c:\windows\system32\inetsrv\clusweb.vbs for the script parameter. Finish.
Right click IIS Resource and Bring Online.
Now right click IIS Resource and select Initiate Failure 3 times in a row. Resources should fail over to Node B.
Verify the website comes up on Node B, and then move the resources back to Node A.
Configure DNS for your new website so that it points to the cluster’s IP address

I specifically created this article for IIS 6 (and not 7) for a couple of reasons. First, that’s the only version I’ve attempted to cluster. Second, Microsoft’s support for clustering on IIS 7 is sketchy at best. Initially when 2008 Server was released, Microsoft announced that they were completely ditching support for clustering IIS. This made sense to most of us, until they shipped Clusweb.vbs and Clusftp.vbs as part of Windows 2008. Those are the same script files that were used to cluster IIS 5.0 and 6.0, so including them seemed a little odd since they were no longer part of a supported configuration. To confuse the issue even further, in just the last few months Microsoft has backtracked on their earlier statements and they now support clustering IIS 7 – without using Clusweb.vbs or Clusftp.vbs – they were shipped with the OS by mistake.

Bottom line – IIS is a platform, not an application, and therefore is not very well suited for clustering. NLB or hardware load balancing should be your first choice, but know that it is possible to cluster IIS if you have no other choice.

Using the Web Deployment Tool to migrate from IIS 6 to IIS 7

Tray Harrison — Wed, 07 Oct 2009 20:23:08 GMT

Install MS Deploy on the source IIS 6 Web server

Install the web deployment tool on the source IIS 6 server. Run the appropriate install package (x86 or x64).
Select Custom for the setup type.
On the Custom Setup page, click the Install the remote agent service down arrow, select Will be installed on local hard drive, then click Next.
Complete the install.
Set the Web Deployment Agent Service to automatic and start it.

Install MS Deploy on the destination IIS 7 Web server

Run the appropriate install package (x86 or x64)
Choose Typical for the setup type
Complete the install

Migrate website

Run the following command on the source (IIS 6) server:
1. msdeploy -verb:sync -source:metaKey=/lm/w3svc/site# -dest:auto,computername=SERVERNAME -enableLink:appPool
Replace site# with the site ID of the website you are trying to migrate. You can find the site ID # by opening Inet Manager, clicking on Web Sites, and view the number listed next to your website in the Identifier column
Replace the value for SERVERNAME with the name of the destination server.
If you need to change the home directory on the destination server, you will need to use a replace rule. Below is a fully functional example. In this scenario I am migrating the site Prismstaging.stewart.com from an IIS 6 box to the new web server SERVER05D. Since the old server has a D: drive and SERVER05D has an E: drive, I need to use the replace rule to change the home directory on the destination.
msdeploy -verb:sync -source:metaKey=/lm/w3svc/43148251 -des:auto,computername=SERVER05D -enableLink:appPool -replace:objectname=metaproperty,scopeattributename=name,scopeAttributeValue=Path,targetAttributename=value,match=d:\\test.website.com,replace=e:\test.website.com

Synchronizing a site between two IIS 7 web servers

Tray Harrison — Fri, 31 Jul 2009 21:21:03 GMT

When working in a web farm environment, keeping content and configuration in sync is a critical piece of the puzzle. With IIS 5 and IIS 6 you could accomplish this with Microsoft’s Application Center. The product certainly has it’s flaws, but once you are able to get it working it is fairly low maintenance. To this day we still have a couple of production farms running App Center and it’s not something I ever have to worry about.

Unfortunately a few years ago you started hearing that Microsoft was discontinuing any future updates or releases of the App Center product. To make matters worse, they kept silent on what would replace the functionality that it provided and many people relied on. That’s where the Web Deployment Tool comes in . While it still doesn’t offer all of the functionality that App Center did, it’s a good start and they are constantly adding new features to it. Recently an article was posted which compares the features offered in both App Center and the Web Deployment tool.

What I was looking to do was use the Web Deployment Tool, and specifically the command line tool within it called MSDeploy.exe, to be able to synchronize content and all IIS configuration for a website between a source and destination server. To do that, follow these steps:

Download the latest version of the Web Deployment tool from http://www.iis.net/extensions/WebDeploymentTool . Make sure you get the correct version for your environment – x86/x64
Run the install on each server that you want to synchronize, choosing Complete to install all components
Create a folder on your server to store the deployment scripts in. I use E:\WebsiteSyncJobs
Create a folder within there for each website that you want to synchronize.
Create a batch file using the info below as a template. Edit the values to match your website and server names. For the apphostconfig value, replace sample.website.com with the URL of the website you’ll be synchronizing. For the computername value, replace SERVERNAME with the name of the destination server that you will be synchronizing with. The below script will create an IIS backup prior to doing anything, and then move over your website and content and write everything to a log.

@ Echo Off
ECHO Starting Deployment on %DATE% at %TIME% >>msdeploySync.log
ECHO Starting Deployment on %DATE% at %TIME%
%windir%\system32\inetsrv\appcmd add backup
"C:\Program Files\IIS\Microsoft Web Deploy\msdeploy.exe" -verb:sync -source:apphostconfig="sample.website.com" -dest:apphostconfig="sample.website.com",computername=SERVERNAME -enablelink:AppPoolExtension >>msdeploysync.log
ECHO Deployment Complete on %DATE% at %TIME% >>msdeploysync.log
ECHO Deployment Complete on %DATE% at %TIME%
ECHO --------------------- >>msdeploysync.log
pause

Save the file with a name that will make it easy to identify. For example, test.website.com_to_SERVER01.bat
Create your website now on the source server if you haven’t already done so.
Run the batch file. Once it is done, open up the msdeploySync.log file and check for any errors.

If everything is good and there were no issues then the site should now show up on the destination server. You should run the sync script anytime you make content or configuration changes to the website, that way everything will always match up. You could also easily schedule the synchronizations using Task Scheduler.

Implementing Memory Limit Application Pool Recycles for SharePoint Web Applications – Just Do It

Tray Harrison — Thu, 26 Mar 2009 20:49:49 GMT

For the last month or so, we’ve been seeing an increasing number of instances in which our heavily used SharePoint intranet sites were becoming un-responsive due to out of memory errors. On 3/12 I modified the application pool configurations for two of these sites so that they would automatically recycle once they consumed 1300 MB of virtual memory. I’ve found that 1300 MB is the sweet spot and you’ll usually start experiencing stability issues with the worker process if you get much further than that. Keep in mind that even though the virtual memory limit is 2GB for a user mode process on a 32-bit system, you’ll start having problems long before you actually hit the ceiling of that limit.

This change has only been in place for about 2 weeks, but already we are seeing significant improvements in the stability of the applications. Referencing the event logs for Event ID 1073, I found that just between the dates of 2/26 – 3/11 (prior to the change) we had 18 out of memory errors. Here are our stats since setting the virtual memory limits on 3/12:

56 automatic app pool recycles – These are graceful recycles which have zero negative impact to the end user
0 out of memory errors
0 support calls from users stating that the intranet sites were down

You’ll want to continuously monitor the recycle stats and adjust the limits as needed, but making this simple change can go a long way towards making your life easier as an administrator and keeping your customers happy.

FIX: You receive a 401.1 error when browsing a web site directly from the server using Integrated Authentication

Tray Harrison — Wed, 25 Mar 2009 17:49:54 GMT

Today I came across an interesting issue that had me pulling my hair out (well it would have if I had any). I was troubleshooting a new internal web site which uses integrated security instead of anonymous. When browsing the site from my laptop, I was able to access it with no issues. But if I tried to hit it within a browser on the server or browsed directly from IIS, I encountered a login prompt that I couldn’t get past. Even with my domain admin credentials I could not get past the login prompt, and after three attempts I was met with a 401.1 which IIS told me was due to invalid credentials. But if my credentials were really invalid, why would it work just fine from my laptop with those same credentials?

After looking in the event logs, I found multiple entries for event ID 537 in the security log. Notice the odd characters listed for the Logon Process.

After doing quite a bit of digging through Google, I finally came across this article which explains that this behavior is actually caused by a security feature that was introduced with SP1 for Windows Server 2003. The cause of the errors is a loopback security check that was introduced in order to prevent reflection attacks. With this loopback check in place, you will be unable to authenticate to any site using windows authentication locally from the server.

Moving forward you have a couple of different options. You could perform all testing from somewhere other than the server so that you’ll never run into this issue. Personally, I like to be able to test my web sites directly from the server for various reasons which make the troubleshooting process a little easier in my opinion (detailed error messages, debugging, etc). In our environment, we never use Windows Authentication anywhere but our internal network which is well protected from outside users attempting to perform malicious attacks. Therefore I decided to just disable this security feature for the particular server I was troubleshooting this on. To do that, make the following changes to the registry and reboot the server to put them into effect.

Click Start, click Run, type regedit, and then click OK.
In Registry Editor, locate and then click the following registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
Right-click Lsa, point to New, and then click DWORD Value.
Type DisableLoopbackCheck, and then press ENTER.
Right-click DisableLoopbackCheck, and then click Modify.
In the Value data box, type 1, and then click OK.
Quit Registry Editor, and then restart your computer.

That’s all there is to it. You’ll now be able to browse sites using integrated security directly from the server any time you need to.

A new venture

Tray Harrison — Fri, 09 Jan 2009 20:59:37 GMT

I've been invited to contribute to an exciting new project called Digital Meld . Please check it out as this site provides great information from experts on a wide range of technology. I will continue to update my geek blog from time to time, but for now will focus on assisting with getting D|M off the ground. Please check it out and offer any feedback you can think of.

-Tray

Enabling Windows Authentication on IIS 7

Tray Harrison — Tue, 06 Jan 2009 22:10:46 GMT

Recently I built some new IIS 7 server and realized that I had forgotten to include Windows Authentication as one of the modules during the initial install. No big deal, I'll fire off a ServerManagerCmd command and then it will be available. So I opened a command prompt and ran the following:

ServerManagerCmd.exe -i Web-Windows-Auth

Now in theory, you would think that after doing this and opening up or refreshing Inetmgr that I would now see Windows Authentication as an option for my site. Wrong. What I found was that I actually had to run an old school IIS reset before I could get it to show up. So try restarting IIS if you run into this, it should take care of the issue and allow you to enable Windows Auth on your site.

IIS 5 - How to resolve "The specified handle is invalid"

WorldofTray — Thu, 13 Nov 2008 19:19:55 GMT

This morning I received a request regarding one of our production critical servers that is STILL running on IIS 5 (i know, i know). Apparently IIS had "disappeared" as one of the other techs here put it. Well it didn't exactly disappear, but it sure got hosed.

I logged into the box, and when I attempted to run InetManager I received the following error:

"The specified handle is invalid. Do you want to continue to connect in the future?"

I then noticed that the WWW service wasn't running, so when I attempted to start it I got another not so nice error:

"Could not start the World Wide Web Publishing Service on Local Computer.
Error 1008: An attempt was made to reference a token that does not exist."

Needless to say, errors like this on an IIS 5 server are not how I like to start my day. Now I had actually seen this behavior in the past on other IIS 5 boxes, and after unsuccessfully attempting to restore the metabase, I ended up having to just reinstall IIS and manually reconfigure the apps. This time though, I did a little more research and came across THIS article which described my symptoms to a tee . Note that I am not running SMS 2003 as the article mentions.

The thought of deleting and renaming machine keys made me a little nervous, but then I thought screw it, lets see if this works. I followed the steps in the article exactly and to my great relief the WWW service started right up afterwards! I did run into one problem though once I got IIS back up and running. Apparently messing with the machine keys jacks up your SSL settings, so if you are using SSL on any of your web apps on the affected server, you will probably have to follow these steps like I did.

After doing everything in the article I was able to hit my websites now over http with no problem. But when I attempted to hit the one site using SSL over https I received a "Page Cannot be displayed." I checked out the System log and saw a ton of entries like this:

"A fatal error occurred when attempting to access the SSL server credential private key. The error code returned from the cryptographic module is 0x80090016. "

The fix for this was pretty simple. I just had to open up the remove the cert from my website in IIS, open the Certificates snap-in within an MMC, delete the certificate, and then re-import it and assign it to my site. Luckily I had a copy of the .pfx handy so getting it re-applied was not an issue.

I hope this info has helped someone. Hopefully this will serve as a kick in the ass to our product teams to listen to us when we say we've got to get these apps off of IIS 5 ASAP!