We have a Sharepoint 2010 farm. There are a few pages that come up in SEO search results that I need to quash. How do I go from a URL to the source of the content in my farm? The URL has the form:


I want to hide (possibly delete) anything at level1 or lower.

  • Who are you hiding it from? SharePoint Search Crawler or external crawlers like Google and Bing? – Matthew McDermott Apr 28 '18 at 15:06
  • Sorry I didn't specify. External crawlers. – alphadogg Apr 30 '18 at 15:14

External crawlers are supposed to follow your robots.txt file. There is nothing "forcing" them to do so, but Google, Bing, Yahoo etc all "should". In SharePoint 2010 you create the text file and deploy it to the root of your IIS site. (In 2013/2016 the server does this automatically.)

I always have to play with it but I would start with:

User-agent: *
Disallow: /sites/level1/level2/level3.aspx

Once deployed you can try a validator like http://tool.motoricerca.info/robots-checker.phtml


You "hide" things by managing Site, list/library, folder, item/page permissions.

In your example "level1" is a Site Collection. You delete these from Central Administration. "level2" could be either a subsite or a library. Site Owners and Site Collection Administrators can delete subsites. Site Owners can delete libraries. "level3.aspx" is a web page. It can either be deleted from the library named "level2" or from the root of the site using SharePoint Designer.

If you just want to hide content from search, there is an option in each site's Advanced Settings to exclude the site from search. Libraries have a similar option.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.