Take the 2-minute tour ×
SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. It's 100% free, no registration required.

From the SharePoint Ribbon, for SharePoint 2010, how do we prevent a new blank site from being crawled by Google or any public search engines on the web?



1 Answer 1

up vote 5 down vote accepted

Here are two ways to tell the robots to not crawl your site:

  1. Adding robots.txt with the following content in the root folder with SharePoint Designer (All Items in SPD)

    User-agent: *
    Disallow: /

enter image description here

  1. Adding meta tag in your master page's Head section as below should prevent the robots from crowling you site.

    <meta name="robots" content="noindex" >

I haven't tested it does evey robot out there respects this meta tag setting or not.

It's also posible to prevent only Googlebot from crowling your site as follows

 <meta name="googlebot" content="noindex">

Check this article from google https://support.google.com/webmasters/answer/93710?rd=1

thank you Naim for the rich feedback! But where is the "All Items" folder under my new blank site in SharePoint Designer? I don't see a folder called that...or did mean to just place the robots.txt file in the root directory of that specific site collection? thanks! –  blachawk Feb 3 at 20:25
Yes, when you open a site with SharePoint Designer on the Site Objects you'll see All Items. Just drag and drop the file there. I updated my answer. –  Naim Feb 3 at 20:34
You're the man. thanks –  blachawk Feb 3 at 20:53

This site is currently not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .