Steal This Code

Web site scraping is a great tool, and it is so easy in .Net. But there are some snags, especially if you are trying to build a robust system that can run unattended. One of these problems are web errors like '999 web site temporarily unavailable.' .Net views these errors as fatal and boom you are out of the program. Now you can catch them then recall the mentod, but this recursive action can eat up memory very quickly. There is however another way:

public static string ScrapeURL (string URL) {
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(URL);
HttpStatusCode code = new HttpStatusCode();
HttpWebResponse resp = null;

try {
resp = (HttpWebResponse)req.GetResponse();
code = resp.StatusCode;
} catch(WebException err) {
return null;
}

string strResult;

using (StreamReader sr =
new StreamReader(resp.GetResponseStream())) {
strResult = sr.ReadToEnd();
// Close and clean up the StreamReader
sr.Close();
}
return strResult;
}

Then all you have to do is call this procedure in a do..while (response !=null) loop and away you go. Notice that you avoid stacking up multiple calls because all we are doing here is ignoring the WebErrors.

You can also get more specific and use the code = resp.StatusCode in a case satatement to different things based on the specific HttpWebError. (ie you may want to react to a 404 error differently than you do for a 999 error.

In response to Marcus Eggers' posting, I have added a little info about who I am. Not that I thought he was talking about me specifically, but you never know.....

First off my name is Jeff Tolle. I currently live and work in the Piedmont Triad area of North Carolina. I have many irons in the fire, most of them to do with development using Microsoft Technologies. If you need to get in contact with me, you can use the contact form from this blog. Oh, and Marcus. I am still available to write that article for you. Call me!

Jeff Tolle