Take the 2-minute tour ×
SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. It's 100% free, no registration required.

We are having some issues getting a full crawl to complete by FAST Search connector in our environment. We are attempting to crawl ~24 million records from SQL Server 2008 R2. The crawl process (mssearch.exe) slowly eats up all the memory on the server, and when the memory is completely full it starts using the disc space by storing some files on the Search Service account’s users temp folder and eats up the complete space on the disc… when the server completely run’s out of resource the Mssearch.exe crashes, and restarts itself from the beginning.

Our query environment consists of two SharePoint 2010 Servers w/ 16GB memory each, both set up as crawl components pointing at the same crawl database.

We tried using the Out-Of-the-Box SQL connector and also tried using a custom .NET connector assembly, implemented to perform batching based on the "LastID" filter and an "IDEnumerator". I can see that the connector is batching records properly, so I would expect that the SP Server would stream each batch over to the FAST server as it reads them - however, it still seems to be holding everything in memory or on disc. No connections are being made from mssearch.exe to the FAST server during this time.

What else have we tried?

Custom .NET connector using a "ReadList" operation and batching filters based on BatchId and HasMoreData. Out-of-the-box SQL Connector hitting stored procedures Out-of-the-box SQL Connector hitting the SQL table directly Unfortunately we see the same exact behavior with all of these.

Has anyone out there seen anything similar to this, and have any idea how to get around this?

When we attempted to crawl 4 million records or anything less than that the crawl job completes successfully, One thing I noticed here is that FAST Search connector crawls the complete data and keeps everything in its memory, once it is done with the crawl it then pushes the Index files to FAST server and FAST databases, I expect this pushing happens periodically or in batches so we don’t run out of resources on the SharePoint application server where this crawl component resides.

In our production environment we need to crawl more than 80 million records and there is no way we can increase the resources on the crawl server to hold everything in its memory, In that case I guess it might need more than 120 GB ram and 300 GB of disc space.

Any suggestions will be appreciated, I thank you in advance.

Regards, Rahul

share|improve this question
    
We have enough space on FAST server but do we also need space on SharePoint Application server where the crawl component resides –  Rahul Mar 21 '12 at 16:44
    
are you using the sharepoint BCS connector or the Fast Search jdbc connector? –  Bas Lijten Mar 21 '12 at 18:03
    
We used the BCS connector, we did not try using the jdbc connector. I am not sure but i heard there will few limitations in search administration if we use JDBC Connector, limitations include restrictions in setting the scopes and loosing the ability to manage search from central administration and has to be managed only through command prompt. Please correct me if am wrong. –  Rahul Mar 21 '12 at 18:25
    
there are, indeed, a few limitations. But there are some other advantages over using the BDC: a much higher throughput, more control over incremental updates. The jdbc connector is imho much better for importing large datasets then the BDC. As for using scopes: you can create scopes for Fast, but you need to do this via the API or powershell. The data is stored in another content collection, so you need to create a fast specific scope. The scopes in central admin are "SharePoint scopes" and are very limited. check this blogpost on how to use it tinyurl.com/7x5vuu3 –  Bas Lijten Mar 22 '12 at 7:42
    
Thank you for the detailed explanation Mr. Bass, i will try to implement this crawl using the JDBC connector and will update the test results... fingers crossed hope the crawl runs smoothly. one quick question in order to use JDBC connector we need to use the FAST side crawler... instead of SharePoint connector crawler right. –  Rahul Mar 23 '12 at 13:54
show 1 more comment

3 Answers 3

Rahul,

to answer your question, based on the comments in your question: You should try to crawl and index the items via the JDBC connector instead of the OOTB SharePoint connector. take note that you need to install the java runtime. You will also need a sql database (no express version) when indexing more then 2 million items.

see http://technet.microsoft.com/en-us/library/ff381259.aspx for guidance on configuring the jdbc connector.

share|improve this answer
add comment

Implement a protocol handler with a RootFinder/SpecificFinder - then the batching should work as expected.

share|improve this answer
add comment

I'm not a FAST search expert but I'm pretty sure that you need more disk space on your index servers as there needs to be at least as much room as there is data being indexed. I.e. if you are indexing 500gb of data then you need at least 500gb of space available on the index servers.

Perhaps this might help as well

share|improve this answer
    
Thats not correct, FAST puts at most 40% of the indexed documents to the index. –  Max Melcher Jun 23 '13 at 21:46
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.