For the last month or so, we’ve been seeing an increasing number of instances in which our heavily used SharePoint intranet sites were becoming un-responsive due to out of memory errors. On 3/12 I modified the application pool configurations for two of these sites so that they would automatically recycle once they consumed 1300 MB of virtual memory. I’ve found that 1300 MB is the sweet spot and you’ll usually start experiencing stability issues with the worker process if you get much further than that. Keep in mind that even though the virtual memory limit is 2GB for a user mode process on a 32-bit system, you’ll start having problems long before you actually hit the ceiling of that limit.
This change has only been in place for about 2 weeks, but already we are seeing significant improvements in the stability of the applications. Referencing the event logs for Event ID 1073, I found that just between the dates of 2/26 – 3/11 (prior to the change) we had 18 out of memory errors. Here are our stats since setting the virtual memory limits on 3/12:
- 56 automatic app pool recycles – These are graceful recycles which have zero negative impact to the end user
- 0 out of memory errors
- 0 support calls from users stating that the intranet sites were down
You’ll want to continuously monitor the recycle stats and adjust the limits as needed, but making this simple change can go a long way towards making your life easier as an administrator and keeping your customers happy.
Today I came across an interesting issue that had me pulling my hair out (well it would have if I had any). I was troubleshooting a new internal web site which uses integrated security instead of anonymous. When browsing the site from my laptop, I was able to access it with no issues. But if I tried to hit it within a browser on the server or browsed directly from IIS, I encountered a login prompt that I couldn’t get past. Even with my domain admin credentials I could not get past the login prompt, and after three attempts I was met with a 401.1 which IIS told me was due to invalid credentials. But if my credentials were really invalid, why would it work just fine from my laptop with those same credentials?
After looking in the event logs, I found multiple entries for event ID 537 in the security log. Notice the odd characters listed for the Logon Process.
After doing quite a bit of digging through Google, I finally came across this article which explains that this behavior is actually caused by a security feature that was introduced with SP1 for Windows Server 2003. The cause of the errors is a loopback security check that was introduced in order to prevent reflection attacks. With this loopback check in place, you will be unable to authenticate to any site using windows authentication locally from the server.
Moving forward you have a couple of different options. You could perform all testing from somewhere other than the server so that you’ll never run into this issue. Personally, I like to be able to test my web sites directly from the server for various reasons which make the troubleshooting process a little easier in my opinion (detailed error messages, debugging, etc). In our environment, we never use Windows Authentication anywhere but our internal network which is well protected from outside users attempting to perform malicious attacks. Therefore I decided to just disable this security feature for the particular server I was troubleshooting this on. To do that, make the following changes to the registry and reboot the server to put them into effect.
- Click Start, click Run, type regedit, and then click OK.
- In Registry Editor, locate and then click the following registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
- Right-click Lsa, point to New, and then click DWORD Value.
- Type DisableLoopbackCheck, and then press ENTER.
- Right-click DisableLoopbackCheck, and then click Modify.
- In the Value data box, type 1, and then click OK.
- Quit Registry Editor, and then restart your computer.
That’s all there is to it. You’ll now be able to browse sites using integrated security directly from the server any time you need to.
I've been invited to contribute to an exciting new project called
Digital Meld . Please check it out as this site provides great information from experts on a wide range of technology. I will continue to update my geek blog from time to time, but for now will focus on assisting with getting D|M off the ground. Please check it out and offer any feedback you can think of.
-Tray
Recently I built some new IIS 7 server and realized that I had forgotten to include Windows Authentication as one of the modules during the initial install. No big deal, I'll fire off a ServerManagerCmd command and then it will be available. So I opened a command prompt and ran the following:
ServerManagerCmd.exe -i Web-Windows-Auth
Now in theory, you would think that after doing this and opening up or refreshing Inetmgr that I would now see Windows Authentication as an option for my site. Wrong. What I found was that I actually had to run an old school IIS reset before I could get it to show up. So try restarting IIS if you run into this, it should take care of the issue and allow you to enable Windows Auth on your site.
This morning I received a request regarding one of our production critical servers that is STILL running on IIS 5 (i know, i know). Apparently IIS had "disappeared" as one of the other techs here put it. Well it didn't exactly disappear, but it sure got hosed.
I logged into the box, and when I attempted to run InetManager I received the following error:
"The specified handle is invalid. Do you want to continue to connect in the future?"
I then noticed that the WWW service wasn't running, so when I attempted to start it I got another not so nice error:
"Could not start the World Wide Web Publishing Service on Local Computer.
Error 1008: An attempt was made to reference a token that does not exist."
Needless to say, errors like this on an IIS 5 server are not how I like to start my day. Now I had actually seen this behavior in the past on other IIS 5 boxes, and after unsuccessfully attempting to restore the metabase, I ended up having to just reinstall IIS and manually reconfigure the apps. This time though, I did a little more research and came across
THIS article which described my symptoms to a tee . Note that I am not running SMS 2003 as the article mentions.
The thought of deleting and renaming machine keys made me a little nervous, but then I thought screw it, lets see if this works. I followed the steps in the article exactly and to my great relief the WWW service started right up afterwards! I did run into one problem though once I got IIS back up and running. Apparently messing with the machine keys jacks up your SSL settings, so if you are using SSL on any of your web apps on the affected server, you will probably have to follow these steps like I did.
After doing everything in the article I was able to hit my websites now over http with no problem. But when I attempted to hit the one site using SSL over https I received a "Page Cannot be displayed." I checked out the System log and saw a ton of entries like this:
"A fatal error occurred when attempting to access the SSL server credential private key. The error code returned from the cryptographic module is 0x80090016. "
The fix for this was pretty simple. I just had to open up the remove the cert from my website in IIS, open the Certificates snap-in within an MMC, delete the certificate, and then re-import it and assign it to my site. Luckily I had a copy of the .pfx handy so getting it re-applied was not an issue.
I hope this info has helped someone. Hopefully this will serve as a kick in the ass to our product teams to listen to us when we say we've got to get these apps off of IIS 5 ASAP!
Today I finally decided to play around with LogParser after planning to get familiar with it for the last few years. This is one seriously powerful little tool! Already I've found several uses for it as a web analytics tool, as I've been able to do more with LogParser in a few hours than I've ever been able to do with our (purchased) web analytics tool which shall remain nameless. It's also come in handy already as helping me to track down a runaway process that was chewing up disk space at an alarming pace. I see a bright future between myself and LP
.
Download Logparser 2.2:
www.microsoft.com/technet/scriptcenter/tools/logparser/default.mspx
Not too long ago I decided to look into a quick and easy solution for our corporate web team to use for generating self signed SSL certs on our dev/test web apps. I had a vague memory that there was something included in the
IIS Resource Kit that would do just that, so I decided to check out SelfSSL.
SelfSSL is a command line based tool that is very straightforward and you can figure out everything you need to know from the included examples. Initially it seemed to be exactly what I was looking for. That is until we started to implement this on more and more sites, and started to notice a disturbing trend. I'd generate a cert for a new site, which would work, but in turn would break the SSL on an existing site. After this happened for about the 10th time I finally got smart and decided to do some research on the issue. Sure enough with a little help from something I like to call Google, I had found my answer in about 5 seconds.
What I discovered is that there is a known bug in the version of SelfSSL included with the resouce kit which causes it to crash and burn if you try to use it on more than one site. The good news is that I found
THIS post on David Wang's blog which says the bug has been fixed in the updated SSL Diag 1.1 version. The bad news is Microsft knows about the bug, yet doesn't bother to fix it in the resource kit. Why keep a resource kit with known bugs available for download? Come on MS, you're better than that!