Take the 2-minute tour ×
SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. It's 100% free, no registration required.

We are having some issues getting a full crawl to complete in our FAST Search evironment. We are attempting to crawl ~24 million records from SQL Server 2008 R2. The crawl process (mssearch.exe) slowly eats up all the memory on the server, crashes, and restarts itself from the beginning.

Our query environment consists of two SharePoint 2010 Servers w/ 16GB memory each, both set up as crawl components pointing at the same crawl database.

The connector we are trying to use is a custom .NET connector assembly, implemented to perform batching based on the "LastID" filter and an "IDEnumerator". I can see that the connector is batching records properly, so I would expect that the SP Server would stream each batch over to the FAST server as it reads them - however, it still seems to be holding everything in memory. No connections are being made from mssearch.exe to the FAST server during this time.

What else have we tried?

  • Custom .NET connector using a "ReadList" operation and batching filters based on BatchId and HasMoreData.
  • Out-of-the-box SQL Connector hitting stored procedures
  • Out-of-the-box SQL Connector hitting the SQL table directly

Unfortunately we see the same exact behavior with all of these.

Has anyone out there seen anything similar to this, and have any idea how to get around this?

IDEnumerator code:

public static IEnumerable<string> GetItemIds(string LastSeenId)
    {
        // The current batch ID
        int batchId = 0;

        // Parse the current batch id from the 'LastSeenID'.  We are using LastSeenId as 
        // a batch (or page) number instead of a true 'ID', since we have a 3-field
        // composite key and wanted to quickly try out this method of batching.
        if (!string.IsNullOrEmpty(LastSeenId))
        {
            string[] idStrSplit = LastSeenId.Split(new char[] { '|' });
            int.TryParse(idStrSplit[3].ToString(), out batchId);
        }
        batchId = batchId + 1;

        List<string> idList = new List<string>();

        // Build a list of IDs from the SQL table
        using (SqlConnection conn = new SqlConnection("<connection string>"))
        {
            conn.Open();

            using (SqlCommand cmd = new SqlCommand("spDocQGetListByBatchId", conn))
            {
                cmd.CommandType = System.Data.CommandType.StoredProcedure;
                cmd.Parameters.AddWithValue("BatchId", batchId);
                cmd.Parameters.AddWithValue("BatchSize", 25000);

                using (SqlDataReader dr = cmd.ExecuteReader())
                {
                    // First get the current set of results
                    while (dr.Read())
                    {
                        // Convert our composite key into something that we can easily
                        // pass into the ReadItem calls.
                        string id = string.Format("{0}|{1}|{2}|{3}",
                            dr["MHN"],
                            dr["EventDt"],
                            dr["Accn"],
                            batchId);

                        idList.Add(id);
                    }
                }
            }
        }

        return idList;
    }
share|improve this question
    
An interesting scenario indeed. Are you using a SqlReader to feed the data? A bit hard to pinpoint any trouble without seeing the code. And, you could look into using the FAST Database Connector instead - technet.microsoft.com/en-us/library/ff381259.aspx. –  Mikael Svenson Jan 6 '12 at 19:56
    
Thanks Mikael. Yep, we are using SqlDataReader to feed data. I left out code since we are seeing this with the OOB database connector as well, but I just attached our IdEnumerator method. It is using LastID in a hacky manner, but it does seem to batch the records correctly. I will do some more looking into the JDBC connector you linked to. We spoke to a consultant about that, but they said that it removed support for result profile pages and bypasses all of the OOB configuration through Central Admin, so we avoided it without much thought. –  MattG Jan 6 '12 at 22:55
    
On another note in case it comes up - I duplicated our SQL table and changed it to use an int primary key instead of a composite one, just in case. I had read some things that made me feel like it may have an effect for some reason, but no luck there. –  MattG Jan 6 '12 at 22:59
add comment

1 Answer 1

If you change your code to use "yield return" you should give out id's as the crawler asks for it, instead of building up a long list internally which is then returned. I haven't tried this myself, but using yield is the perfect way to fetch data as needed. Could be worth a try.

I suspect you can do something similar when it ask for each item, if you keep a SqlDataReader in the back and caches items, and return them as they are asked for, insteda of doing a separate call per item.

Btw, are you getting a crash if you index the same data to the internal SP search?

using (SqlDataReader dr = cmd.ExecuteReader())
{
    // First get the current set of results
    while (dr.Read())
    {
        //idList.Add(id);
        yield return id;
    }
}
share|improve this answer
    
This sounded promising... I implemented the 'yield return' this morning and kicked off a crawl - unfortunately it ended in the same result. Crash aside though, I think your 'yield' approach is definitely the right way to go here. I haven't tried indexing the data into the internal SP search - I'll give that a shot and see what happens. –  MattG Jan 7 '12 at 20:13
    
@MattG Any progress on this? –  Mikael Svenson Jan 23 '12 at 10:32
    
Unfortunately not yet. Delivered a pile of logging and trace info to our Microsoft support rep last week, and currently trying to replicate the issue using a CloudShare environment to make sure it isn't just our server configuration. I'll update here if any resolution is found. –  MattG Jan 23 '12 at 15:01
    
I put together a quick farm on CloudShare and got surprising results... Crawling a SQL table w/ 22 million records, total size of ~15GB. Mssearch.exe hovers around 1GB memory usage, and 'successes' are slowly trickling in. Differences in this setup? SP1, local SQL Server, one crawl node (shouldn't make things better). The SQL server is taking up 6.5 of the 8GB on the server, and still working (though really slow). Will have to try applying SP1 to our environment. –  MattG Jan 25 '12 at 15:22
    
Nevermind, admin recently pushed us to SP1... hmm. –  MattG Jan 25 '12 at 19:55
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.