Search only crawls a minimal part of my site

Question

I have a regular SharePoint site setup that is setup to allow anonymous access. It has around ~40 pages setup with various text and pictures, spread over 6 sub-webs.

Crawling is setup, the content account is given access to the site, but still only 5 pages seem to be crawled. The crawl log reveals a few pages that have been excluded due to crawl rules (*_vti_bin and similar), but none of the missing pages are mentioned.

The navigation and each sub-web frontpage is setup to ensure there is a link to every page I would want crawled, but they are still not included.

Does anyone have any ideas as to what is causing this? What are the "usual suspects" for cases like this?

Answer 1

Make sure your content source is set so that it crawls your content as a SharePoint site and not a web site (via http). The latter will only crawl pages that have direct links to them, which could be an issue if you have content that is not linked on the site.

It is also worth checking your Crawl Rules as well since they can sometimes filter out content that you do not intend to be removed. Remember that when Search is about to index a URL, it processes the rules until the very first match in the list. If that match is an exclude, then the content is excluded - even if there is another rule further down that would be an include. The Crawl Rules page has a spot where you can paste in a URL to test it and see which rules is picking it up.

Answer 2

Start by checking all of your sites configuration, and see if they are allowed for indexing you can do this by entering your site settings and the choose "Search and Offline Availability", then proceed to check your libraries, after you do this, run a full crawl check the crawl log for any errors, and try your search again.