Arrange content source by content database

Question

As per this question when we move site collections around, they drop from the search results. Our last full crawl took over 8 days and while that's happening we don't get incremental results which doesn't make for happy users.

We have around 100,000 site collections distributed across 21 content databases with nearly a terrabyte of data. The site collections are basically self contained projects. When the project is over, we archive the site collection to an archive content database so that the documents are still available and searchable. Unfortunately, when we move a site collection (using stsadm) all the items in it drop from the search index.

So I have been thinking about breaking the site up into content sources, one for each content database. Then I can incrementally do full crawls of the various content database over a week or two without losing all my incremental search results for the other content sources. However, having a content source crawl a content database isn't an option in Sharepoint 2007 content sources.

The live databases have around 2,000 site collections and the archive content databases have around 10,000 site collections. I could get a list of site collections in a content database and have a content source that uses that list but over 10,000 entries might be a little much.

Anyone got some good ideas about distributing the search load?

Answer 1

Having separate content sources is a good idea. You need this if you want to manage your crawl schedules separately. The recommended limits are 100 start addresses per content source, and 50 content sources per search application.

Rather than add lots of start URL's to your content sources, which would be difficult to manage and take you outside the supported limits, you might find it easier to create a dynamic page that contains links to all the site collections in your content database and use that as your start address for the content source.

10,000 site collections per content database sounds a little high (recommended maximum is 5000).