From the SharePoint Ribbon, for SharePoint 2010, how do we prevent a new blank site from being crawled by Google or any public search engines on the web?



Here are two ways to tell the robots to not crawl your site:

  1. Adding robots.txt with the following content in the root folder with SharePoint Designer (All Items in SPD)

    User-agent: *
    Disallow: /

enter image description here

  1. Adding meta tag in your master page's Head section as below should prevent the robots from crowling you site.

    <meta name="robots" content="noindex" >

I haven't tested it does evey robot out there respects this meta tag setting or not.

It's also posible to prevent only Googlebot from crowling your site as follows

 <meta name="googlebot" content="noindex">

Check this article from google https://support.google.com/webmasters/answer/93710?rd=1

  • thank you Naim for the rich feedback! But where is the "All Items" folder under my new blank site in SharePoint Designer? I don't see a folder called that...or did mean to just place the robots.txt file in the root directory of that specific site collection? thanks!
    – klewis
    Feb 3 '14 at 20:25
  • Yes, when you open a site with SharePoint Designer on the Site Objects you'll see All Items. Just drag and drop the file there. I updated my answer. Feb 3 '14 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.