I got the following warning after I crawled a site. Info from the Crawl Log:

The content for this address was excluded by the crawler because this item was marked with a no-index meta-tag. To index this item, remove the meta-tag and recrawl.

Sound easy but I don't no where to do this.

In the Crawl Configuration section I have selected "Include all items in this path". Also the Content Access Account has full permissions on content to be crawled.

6 Answers 6


After doing a lot of matrix testing on some troublesome URLs, there appear to be at least three things which generate this same message.

1) There is, in fact, a bot tag in a page which is having its content crawled.
2) Your SharePoint SSA is not configured to crawl the file type reflected in the URL. We are using a custom protocol handler to crawl in database information and then pass a destination URL. (Imagine that we were crawling a client list, and then making www.your_big_client.com as the destination when you searched the client name.) In our case, even though the actual URL was not being crawled, SharePoint was parsing the URL to see if it was of a valid document extension – in this case ‘.wl’. In some of my dummy URL’s, I was feeding in .asmx files and these were generating the same error. You will need to go to File Types in the SSA, and add your oddball extensions. You may need to restart the Search Service.
3) A crawl rule is in place which prevents crawling. You may specify crawl rules in order to avoid crawling particular domains, machines, page names, etc. In previous versions of SP, this had its own distinctive message, but I see evidence that crawl rule exclusions now trigger the above message. Go to Crawl Rules in your SSA, and input the offending URL to see if it matches an exclusion rule.


I had the same issue, fixed it by publishing and approving the homepage (publication site)


Have you placed some kind of robots.txt in the root of the website? Don't know if Sharepoint respects the robots file (or if it would show this error when it has), but it's worth checking out.

It could be there's some http-module which places the no-index tag in the header of the files, but that's a long-shot. Don't think that would go unnoticed.


Try going to site settings -> search visibility for the site in question and make sure that "Allow this web to appear in search results" is set to yes.

  • Hi Steve, Thanks for your response. I have checked above mentioned settings on MOSS 2007 site and it is set to yes. When i start crawl it does not crawl the content ad gives warning message as "The content for this address was excluded by the crawler because this item was marked with a no-index meta-tag. To index this item, remove the meta-tag and recrawl" I would like to know one thing can we crawl MOSS 2007 server content from Sharepoint 2010 server???
    – user1168
    Commented Jul 31, 2010 at 12:38
  • I don't know if you can or not. I haven't tried yet. I would also check your crawl rules and make sure the URL is allowed. Commented Jul 31, 2010 at 13:12

The SharePoint crawl log should have given you the URL of the item it emitted the warning for.

You should check if the page contains a meta no index tag.



Try checking File types to crawl, I had excluded aspx files and was getting the same error when crawling profiles

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.