I notice in my Crawl Log I am seeing entries like this:

The content for this address was excluded by the crawler because this item was marked with a no-index meta-tag. To index this item, remove the meta-tag and recrawl.

Mostly these are on directories, like: http://localhost/planning/planningfor2011/ . All of our content is searchable, and I get crawl data from other areas, but these directories seem to be having issues, far as I can tell we don't have any meta-tags on the directory level and we don't on documents underneath. We do have publishing turned on in the Library, would that cause an issue with crawling the top level directory or is there some site setting we might need to change on the top level? Or are these just a distraction and I can ignore them?

Our sites are set to crawl Local SharePoint sites, no crawl rules to eliminate these directories and the account has the right permissions to crawl the Library. We are using SP2010 Enterprise with Search Server.

Addendum: All Site Settings and Library Settings are set to have the pages and sub-sites included in Search, I've gone over the settings multiple times and there is nothing blocking that I can see in the UI.

More information: So we've got some crawls that go but all they do is crawl the top level of the sites, if there are pages in a site those never get seen. Results only show the top level pages and nothing else. We've reindexed and recrawled but can't get additional pages in the index.

2 Answers 2


So it turns out having multiple rules to exclude specific directories were causing crawls to stop doing depth, once those were removed we ended up with Crawls that worked. I still don't understand why excluding some of the Forms directories caused this problem but what we ended up with as a base included the following only:

  • http://*_layouts/* - exclude
  • http://*spsdisco.aspx - exclude
  • http://dispform.aspx - exclude
  • http:// - include

Trying to clear out other errors that appeared in the log generated no crawling, hopefully this will help someone else getting this issue.


Make sure that the "Allow this web to appear in search results" option is set on each site and that the "Allow Items in this document library to appear in Search Results" option is set on the document libraries.

  • Yes, all settings show that they should appear. There appears to be nothing blocking them in the Site Settings or Library Settings.
    – MichaelF
    May 2, 2011 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.