The crawler crawls everything successfully except the below URL http://localhost/alums/lists/324324_classnotes/brown,%20harold%20jr.%20'74 where it gives an error in the crawl log like "Content for this URL is excluded by the server because a no-index attribute".

I do not have any crawl rules setup for any of my content sources.

Any ideas?

Thanks, Michael

Update: I checked the source of the page and I found out below tag in it. <META NAME="ROBOTS" CONTENT="NOHTMLINDEX"/>

Is it possible if this is blocking the search to work?

  • See Anders' comment on the Crawl Rules for the site (In Site Settings->Search Visibility).
    – James Love
    Commented Aug 27, 2010 at 16:45

2 Answers 2


Check the source of the page and see if theres a

<META name="robots" content= "noindex">

Tag in the header. This could be set by the <SharePoint:RobotsMetaTag ID="RobotsMetaTag1" runat="server" /> in the master page.

Have you checked the crawl rules both for the site (Search visibility) and for the pages library (index settings on doclib)?


in crawl rules untick,

"Follow links on the URL without crawling the URL itself"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.