For what reasons would a site be excluded from a full crawl?

Question

This is what I've checked so far:

• The default page of the site is published
• In site admin it is set to 'allow this site to appear in search results'
• In site admin of the root site, it will search all web parts
• Everything builds and can be browsed as expected (no bugs/404's etc.)

Still no luck. Is there something else I'm missing? Or any similar scenarios you've came across?

Answer 1

What about your search crawl rules?

Does the full crawl log say that it's excluded or does the log say that it indexed it but it doesn't show up in search results?

Answer 2

If you have a lot of users in the site (usually over ~1800), your site will no longer be crawled. This count includes unique users in sharepoint groups or site permissions. Users in active directory security groups don't count towards the limit (the security group counts as 1).

If this were the case, you would likely be receiving a message like The parameter is incorrect in your crawl log.

A very good explanation of the issue here: SharePoint Indexing Limited by 64 kB ACL Limit

Answer 3

The most ubiquitous reason that Sharepoint Search/Crawling fails completely is the legendary Loopback Security Check

Answer 4

A few reasons that I'm thinking of

• the targeted site is not part of any content source
• the crawler account that is used has no access to the target site
• A specific robots.txt is available at the root of the web application preventing a proper crawl / indexing
• A too generic crawl rule is defined or wrongly positioned (cascading effect)
• an unconfigured binding / AAM is used to access the site.

The easiest way to understand what's going wrong, if the suggestions above do not help is to install Fiddler on the Crawl server and set the Search Service Application proxy to the one used by Fiddler. That way whenever you start a full / incremental crawl, you'll have a clear view of what's preventing / blocking the crawl.

Hope it helped !