Content for this URL is excluded by the server because a no-index attribute

Question

The crawler crawls everything successfully except the below URL http://localhost/alums/lists/324324_classnotes/brown,%20harold%20jr.%20'74 where it gives an error in the crawl log like "Content for this URL is excluded by the server because a no-index attribute".

I do not have any crawl rules setup for any of my content sources.

Any ideas?

Thanks, Michael

Update: I checked the source of the page and I found out below tag in it. <META NAME="ROBOTS" CONTENT="NOHTMLINDEX"/>

Is it possible if this is blocking the search to work?

Answer 1

Check the source of the page and see if theres a

<META name="robots" content= "noindex">

Tag in the header. This could be set by the <SharePoint:RobotsMetaTag ID="RobotsMetaTag1" runat="server" /> in the master page.

Have you checked the crawl rules both for the site (Search visibility) and for the pages library (index settings on doclib)?

Answer 2

in crawl rules untick,

"Follow links on the URL without crawling the URL itself"