Search scopes have too few items

Question

This is a difficult question

I am trying to figure out how SharePoint 2010 Search and Scope works. I have a site that contains 369 sites, and 5563 total pages within those sites. The sites are all publishing sites, so all the pages are within Pages libraries. I've setup 9 custom scopes so that my users can search specific areas of the site, but the number of items in each scope doesn't come close to the total pages

My Content Source is my site's URL, e.g. https://mysite.com, and the Search Administration page, System Status section, shows 98,480 searchable items.

The default All Sites scope shows 21600 items.

Having run a powershell script to get ALL the pages, I see that there are, for example, 851 publishing pages within the site and subsites of https://mysite.com/us, but the scope I setup using this exact URL only shows 102 items. Another example is the site and subsites of https://mysite/support/version2, which has 1218 pages (according to the powershell script), but only 650 items according to the scope using the same URL.

Clearly something is wrong.

Again, my search's content source uses https://mysite.com as the 'Start Address', and I have the following crawl rules

• https://mysite.com/****/Documents/**** (exclude)
• https://mysite.com//_layouts/ (exclude)
• https://mysite.com//_vti_bin/ (exclude)
• https://mysite.com/* (include - crawl content as http pages)

The first three rules are to remove items from the search results that are shown to my users, while the final rule is so that pages within sites that have Anonymous Access turned off can still be seen in search results.

Given all this, is there a reason why my scopes are only showing a fraction of the items they are supposed to?

These rules should not impact the pages within my Pages libraries, should they?

It did seem that before I added the 2nd and 3rd rules (layouts & _vti_bin), that the scopes had way more items in them, but then the 'searchable items' was around 400,000 instead of 98,480, and most of these were what I call 'internal system pages' rather than content pages that we actually want people to see.

I've completely lost on this, and could really use some advice - anything!

Answer 1

Have your scopes been "compiled" (updated)? Check the scopes update status in Search Administration.

The second thing to check here is that the pages are in your index. Check your crawl logs, or search for specific pages using the All Sites scope. Perhaps the crawl rules have prevented the crawler from finding your pages - if you set it to crawl as http it will only find pages that are ultimately linked to one of the start addresses.

If there is a page that you can't find in the index, but should be there, enter its URL in the crawl rules configuration page to see if one of the crawl rules is excluding it. If the page is in the index and the scopes are all updated then there must be a problem with the scope rule.