Geeks With Blogs

News
+Jeff Julian GWB Tag







SharePoint Consulting (Kansas City Metro)

SharePoint Training

AJI Software - Careers





Check out some of my wife's products!:

Cake Mixes - Made With Scratch
Cookie Mixes - Made With Scratch
Marinades - Made With Scratch

Archives
Jeff Julian's Blog Just Another Geek With A Blog

One of the things I always forgot to add to my web applications is the Robots.txt file that Search Engines use to see what they should index.  This file and site maps help make your site easier to navigate by the bots and let them know what is legal and what you would rather not have the published in their engines.  I typically add any administrative pages or account pages even though they are protected by security, no need for the login page to be index if they sniff the link.

So how do you add Robots.txt to your MVC 3 application?  Glad you asked, here is a little code to get you started.

Sample Code

1.  Select the controller you would like to use for the robots.txt output.  I chose the HomeController in my application as I use it for most “top level” generic links like about us, contact us, index, etc. 

2. Create a method called Robots to handle the request.

#region -- Robots() Method --
public ActionResult Robots()
{
    Response.ContentType = "text/plain";
    return View();
}
#endregion

3. Add the Robots.cshtml view to your Controller’s View directory.  Here is the code I have in my view, yours will vary.

@{
    Layout = null;
}
# robots.txt for @this.Request.Url.Host
 
User-agent: *
Disallow: /Administration/
Disallow: /Account/

4. Load up the class you are using to control your routes, if you are in an Area, this could your AreaRegistration class.  If you are at the top like I am and using the standard MVC template, this is probably the Global.asax.cs file.  Add your route to this file, mine looks like this.

routes.MapRoute("Robots.txt",
                "robots.txt",
                new { controller = "Home", action = "Robots" });

5. Compile and test.

 

Conclusion

If you have an internet facing site, the chances are you will have a bot find you are request this page.  You might as well give them the benefit of the doubt and let them know where you want them to go.  Also you will save yourself some error log when this page is requested and no controller is found.

Just like anything in ASP.NET, there are many ways to solve this riddle, if you use a different approach, please feel free to share it in the comments.

 

Technorati Tags: ,,
Posted on Wednesday, April 25, 2012 7:47 AM | Back to top


Comments on this post: Adding Robots.txt to your ASP.NET MVC 3 Applications

# re: Adding Robots.txt to your ASP.NET MVC 3 Applications
Requesting Gravatar...
Why would you do this over just simply placing a "normal" robots.txt in the site root? Adding a route and view does seem somewhat overkill when the plain text file works fine!

Regards;
Richard Moss
Left by Richard Moss on Apr 25, 2012 8:00 AM

# re: Adding Robots.txt to your ASP.NET MVC 3 Applications
Requesting Gravatar...
I totally agree that if you are not going to need dynamic syntax for site map links or protected directories, keep the static text file. In my case, we often have large amounts of content leading to a variable amount of sitemaps, so being able to dynamically generate the robots.txt helps.
Left by Jeff Julian on Apr 25, 2012 8:17 AM

# re: Adding Robots.txt to your ASP.NET MVC 3 Applications
Requesting Gravatar...
Hey Jeff,

Nice one!

I had to enable RAMFAR so that the route could be caught instead of being handled by the staticfilehandler. It was giving me a 404 since there's no static file in the game (it's dynamically created):

<system.webServer>
<modules runAllManagedModulesForAllRequests="true" />
...

Look here for the details: http://stackoverflow.com/q/11257768/114029


Thanks,

Leniel
Left by Leniel Macaferi on Aug 06, 2012 11:04 AM

# re: Adding Robots.txt to your ASP.NET MVC 3 Applications
Requesting Gravatar...
Thanks, this was very useful. Do you have any tutorial for creating sitemap for MVC 3 web application. Content i am looking at is very dynamic.
Cheers
Left by Anish on Aug 10, 2012 9:02 PM

# re: Adding Robots.txt to your ASP.NET MVC 3 Applications
Requesting Gravatar...
I thought it was useful. A dynamic robots.txt was very helpful, since we have to deploy to several servers and need to map the sitemap to the full URL. Using robots.txt means you don't have to edit for everyone server, local, dev, test, staging, production.
Left by Ken on Oct 24, 2012 11:43 AM

# re: Adding Robots.txt to your ASP.NET MVC 3 Applications
Requesting Gravatar...
Thanks, it very usefull!
Left by ppl on Feb 21, 2014 5:48 PM

Your comment:
 (will show your gravatar)


Copyright © Jeff Julian | Powered by: GeeksWithBlogs.net | Join free