I notice in my Crawl Log I am seeing entries like this:
The content for this address was excluded by the crawler because this item was marked with a no-index meta-tag. To index this item, remove the meta-tag and recrawl.
Mostly these are on directories, like: http://localhost/planning/planningfor2011/ . All of our content is searchable, and I get crawl data from other areas, but these directories seem to be having issues, far as I can tell we don't have any meta-tags on the directory level and we don't on documents underneath. We do have publishing turned on in the Library, would that cause an issue with crawling the top level directory or is there some site setting we might need to change on the top level? Or are these just a distraction and I can ignore them?
Our sites are set to crawl Local SharePoint sites, no crawl rules to eliminate these directories and the account has the right permissions to crawl the Library. We are using SP2010 Enterprise with Search Server.
Addendum: All Site Settings and Library Settings are set to have the pages and sub-sites included in Search, I've gone over the settings multiple times and there is nothing blocking that I can see in the UI.
More information: So we've got some crawls that go but all they do is crawl the top level of the sites, if there are pages in a site those never get seen. Results only show the top level pages and nothing else. We've reindexed and recrawled but can't get additional pages in the index.