As per this question when we move site collections around, they drop from the search results. Our last full crawl took over 8 days and while that's happening we don't get incremental results which doesn't make for happy users.
We have around 100,000 site collections distributed across 21 content databases with nearly a terrabyte of data. The site collections are basically self contained projects. When the project is over, we archive the site collection to an archive content database so that the documents are still available and searchable. Unfortunately, when we move a site collection (using stsadm) all the items in it drop from the search index.
So I have been thinking about breaking the site up into content sources, one for each content database. Then I can incrementally do full crawls of the various content database over a week or two without losing all my incremental search results for the other content sources. However, having a content source crawl a content database isn't an option in Sharepoint 2007 content sources.
The live databases have around 2,000 site collections and the archive content databases have around 10,000 site collections. I could get a list of site collections in a content database and have a content source that uses that list but over 10,000 entries might be a little much.
Anyone got some good ideas about distributing the search load?