Jeff Julian's Blog

Just Another Geek With A Blog


+Jeff Julian GWB Tag

SharePoint Consulting (Kansas City Metro)

SharePoint Training

AJI Software - Careers

Check out some of my wife's products!:

Cake Mixes - Made With Scratch
Cookie Mixes - Made With Scratch
Marinades - Made With Scratch

My Stats

  • Posts - 708
  • Comments - 637
  • Trackbacks - 925

Tag Cloud

Recent Comments

Recent Posts

Article Categories


Image Galleries

My Company

My Fav. Sites


One of the things I always forgot to add to my web applications is the Robots.txt file that Search Engines use to see what they should index.  This file and site maps help make your site easier to navigate by the bots and let them know what is legal and what you would rather not have the published in their engines.  I typically add any administrative pages or account pages even though they are protected by security, no need for the login page to be index if they sniff the link.

So how do you add Robots.txt to your MVC 3 application?  Glad you asked, here is a little code to get you started.

Sample Code

1.  Select the controller you would like to use for the robots.txt output.  I chose the HomeController in my application as I use it for most “top level” generic links like about us, contact us, index, etc. 

2. Create a method called Robots to handle the request.

#region -- Robots() Method --
public ActionResult Robots()
    Response.ContentType = "text/plain";
    return View();

3. Add the Robots.cshtml view to your Controller’s View directory.  Here is the code I have in my view, yours will vary.

    Layout = null;
# robots.txt for @this.Request.Url.Host
User-agent: *
Disallow: /Administration/
Disallow: /Account/

4. Load up the class you are using to control your routes, if you are in an Area, this could your AreaRegistration class.  If you are at the top like I am and using the standard MVC template, this is probably the Global.asax.cs file.  Add your route to this file, mine looks like this.

                new { controller = "Home", action = "Robots" });

5. Compile and test.



If you have an internet facing site, the chances are you will have a bot find you are request this page.  You might as well give them the benefit of the doubt and let them know where you want them to go.  Also you will save yourself some error log when this page is requested and no controller is found.

Just like anything in ASP.NET, there are many ways to solve this riddle, if you use a different approach, please feel free to share it in the comments.


Technorati Tags: ,,



Awesome news!  The TechEd 2012 Closing Party is at Universal’s Island of Adventure theme park this year.  The party is on Thursday, which give you a reason not to leave the conference early and actually catch some of the repeat sessions that you missed from crowds. 

I am really excited to check out the Harry Potter area of the park, I have heard great things.  John and I will be making a presence again at TechEd, stay tuned for more details.





I have to admit, I am pretty excited about this.  I am not sure about the dates, but the Oak Park Mall in Overland Park, KS will be getting a Microsoft Store very soon based on this article.   I am guessing this is due to the amount of Best Buy stores that are closing. 

I have never been to a store, but I am excited, what has everyone else thought about the experience?



It has been a week since we moved our System to a new set of load balanced servers and everything has been going great.  I am so amazed at the performance of the new hardware.  On average, we only use less than 5% of the CPU at any given moments or the database and web servers.  I have seen a performance boost in page load as well, but I will have to confirm that with the statistics as they roll in. 

This is all in preparation for a new community we are launching with some friends that we will be announcing shortly.  We will be launching a nice little contest for our bloggers as well.

Technorati Tags: ,