I have a SharePoint 2010 site collection based on a custom Publishing site definition. I'm trying to figure out why sub-sites and pages underneath a certain site (/Foo) are never included in my search results.
There is nothing obvious configured on my /Foo site or its sub-sites that would prevent them from appearing in the search results - the Allow this site to appear in search results setting is most definitely set to Yes. The site and its sub-sites are created using a sub-site Publishing site definition which is also used for all the other sites in the site collection.
Here's a description of my Search setup and configuration in my development environment.
In the configuration of the Search service application:
Content Source
I created a new content source and included http://localhost to the collection of Start Addresses for the content source. I configured a full and incremental search crawl schedule.
Scopes
I created a custom scope and added a rule of type Content Source, and set it to the content source created in the previous step.
In the configuration of the site collection:
Search Settings
I set Site Collection Search Center to Enable custom scopes and set it to the URL of the Search Center: /Search/
I also set Site Collection Search Dropdown Mode to Do not show scopes dropdown, and default to target results page
Search Scopes
In the Search Scopes configuration of the site collection, I configured the custom scope to be the Default Scope under the Search Dropdown and Advanced search groups.
Troubleshooting
Here are things I've looked into and tried while troubleshooting tis:
Tried using the out-of-the-box Local SharePoint Sites content source and defining my scope to only include content from http://localhost.
Examined the crawl logs. None of the sites under /Foo appear in the crawl logs. There are no errors in the log which would indicate that the sites and pages aren't being crawled.
Reset the index and ran a full crawl.
Created an out of the box SharePoint Publishing site and search center and used the default Local SharePoint Sites content source to search for the content. None of the sites under /Foo appear in the results.
Thoroughly stumped; would appreciate any guidance on this.
Edit - 3/37/2012 Ran stsadm -o setproperty -pn portalsuperreaderaccount -pv DOMAIN\user -url http://webappurl to make sure that the search account is also configured as the super reader account. It's a site collection administrator, and also the service account, so I don't think this makes a difference.