We are having some issues getting a full crawl to complete by FAST Search connector in our environment. We are attempting to crawl ~24 million records from SQL Server 2008 R2. The crawl process (mssearch.exe) slowly eats up all the memory on the server, and when the memory is completely full it starts using the disc space by storing some files on the Search Service account’s users temp folder and eats up the complete space on the disc… when the server completely run’s out of resource the Mssearch.exe crashes, and restarts itself from the beginning.

Our query environment consists of two SharePoint 2010 Servers w/ 16GB memory each, both set up as crawl components pointing at the same crawl database.

We tried using the Out-Of-the-Box SQL connector and also tried using a custom .NET connector assembly, implemented to perform batching based on the "LastID" filter and an "IDEnumerator". I can see that the connector is batching records properly, so I would expect that the SP Server would stream each batch over to the FAST server as it reads them - however, it still seems to be holding everything in memory or on disc. No connections are being made from mssearch.exe to the FAST server during this time.

What else have we tried?

Custom .NET connector using a "ReadList" operation and batching filters based on BatchId and HasMoreData. Out-of-the-box SQL Connector hitting stored procedures Out-of-the-box SQL Connector hitting the SQL table directly Unfortunately we see the same exact behavior with all of these.

Has anyone out there seen anything similar to this, and have any idea how to get around this?

When we attempted to crawl 4 million records or anything less than that the crawl job completes successfully, One thing I noticed here is that FAST Search connector crawls the complete data and keeps everything in its memory, once it is done with the crawl it then pushes the Index files to FAST server and FAST databases, I expect this pushing happens periodically or in batches so we don’t run out of resources on the SharePoint application server where this crawl component resides.

In our production environment we need to crawl more than 80 million records and there is no way we can increase the resources on the crawl server to hold everything in its memory, In that case I guess it might need more than 120 GB ram and 300 GB of disc space.

Any suggestions will be appreciated, I thank you in advance.

Regards, Rahul

  • We have enough space on FAST server but do we also need space on SharePoint Application server where the crawl component resides
    – Rahul
    Mar 21, 2012 at 16:44
  • are you using the sharepoint BCS connector or the Fast Search jdbc connector?
    – Bas Lijten
    Mar 21, 2012 at 18:03
  • We used the BCS connector, we did not try using the jdbc connector. I am not sure but i heard there will few limitations in search administration if we use JDBC Connector, limitations include restrictions in setting the scopes and loosing the ability to manage search from central administration and has to be managed only through command prompt. Please correct me if am wrong.
    – Rahul
    Mar 21, 2012 at 18:25
  • there are, indeed, a few limitations. But there are some other advantages over using the BDC: a much higher throughput, more control over incremental updates. The jdbc connector is imho much better for importing large datasets then the BDC. As for using scopes: you can create scopes for Fast, but you need to do this via the API or powershell. The data is stored in another content collection, so you need to create a fast specific scope. The scopes in central admin are "SharePoint scopes" and are very limited. check this blogpost on how to use it tinyurl.com/7x5vuu3
    – Bas Lijten
    Mar 22, 2012 at 7:42
  • Thank you for the detailed explanation Mr. Bass, i will try to implement this crawl using the JDBC connector and will update the test results... fingers crossed hope the crawl runs smoothly. one quick question in order to use JDBC connector we need to use the FAST side crawler... instead of SharePoint connector crawler right.
    – Rahul
    Mar 23, 2012 at 13:54

3 Answers 3



to answer your question, based on the comments in your question: You should try to crawl and index the items via the JDBC connector instead of the OOTB SharePoint connector. take note that you need to install the java runtime. You will also need a sql database (no express version) when indexing more then 2 million items.

see http://technet.microsoft.com/en-us/library/ff381259.aspx for guidance on configuring the jdbc connector.


Implement a protocol handler with a RootFinder/SpecificFinder - then the batching should work as expected.


I'm not a FAST search expert but I'm pretty sure that you need more disk space on your index servers as there needs to be at least as much room as there is data being indexed. I.e. if you are indexing 500gb of data then you need at least 500gb of space available on the index servers.

Perhaps this might help as well

  • Thats not correct, FAST puts at most 40% of the indexed documents to the index. Jun 23, 2013 at 21:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.