Take the 2-minute tour ×
SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. It's 100% free, no registration required.

From the SharePoint Ribbon, for SharePoint 2010, how do we prevent a new blank site from being crawled by Google or any public search engines on the web?

Thanks!

share|improve this question
add comment

1 Answer

up vote 5 down vote accepted

Here are two ways to tell the robots to not crawl your site:

  1. Adding robots.txt with the following content in the root folder with SharePoint Designer (All Items in SPD)

    User-agent: *
    Disallow: /
    

enter image description here

  1. Adding meta tag in your master page's Head section as below should prevent the robots from crowling you site.

    <meta name="robots" content="noindex" >
    

I haven't tested it does evey robot out there respects this meta tag setting or not.

It's also posible to prevent only Googlebot from crowling your site as follows

 <meta name="googlebot" content="noindex">

Check this article from google https://support.google.com/webmasters/answer/93710?rd=1

share|improve this answer
    
thank you Naim for the rich feedback! But where is the "All Items" folder under my new blank site in SharePoint Designer? I don't see a folder called that...or did mean to just place the robots.txt file in the root directory of that specific site collection? thanks! –  blachawk Feb 3 at 20:25
    
Yes, when you open a site with SharePoint Designer on the Site Objects you'll see All Items. Just drag and drop the file there. I updated my answer. –  Naim Feb 3 at 20:34
    
You're the man. thanks –  blachawk Feb 3 at 20:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.