1

Background
My organization has been attempting to use FAST Search for Sharepoint 2010 as a search solution for our DMS. I am currently utilizing a .Net indexing connector to index a database of ~ 8 million records and their accompanying files. The full crawl, albeit time consuming (about 2 weeks), works wonderfully. It is certainly taxing on the crawl and index servers, but well within their limitations.

Problem
The first incremental crawl is causing a memory spike on both the crawl server (mssearch.exe) and index server (sql). I have implemented some custom logging in my connector, and I can see that it gathers all the ids and seems to only process the updated items. However, the incremental crawl never completes and, on the crawl server, mssearch.exe eventually consumes all available resources. On the index server, sqlservr.exe seems to have the same fate. Several full crawls have completed successfully, and services/servers have been restarted multiple times, but I have yet to have any luck. Full and incremental crawls work perfectly when only crawling a portion of the repository (test sizes: 30,000 & 100,000 items).



BCS Connector

  • ReadList, ReadItem, FileStream method
  • Batches files from database based on custom value (tried 100K, 500K, etc)
  • Uses LastModified time stamp field for changes
  • Custom security mapping
  • Custom logging to indicate when batches are returned, when individual entities are processed (based on time interval), and any file stream errors


SharePoint Environment

  • 1x WFE
  • 1x Database Server
  • Application Server (Crawl Server) 12GB Memory
  • 1x Index Server (FAST) 16GB Memory
     public DMSFile ReadItem(string id)
    {
            SqlConnection connection = new SqlConnection(connectionString);
            SqlCommand command = new SqlCommand(cmd, connection);

            DMSFile entity = new DMSFile();

            connection.Open();
            using (SqlDataReader r = command.ExecuteReader())
            {
                while (r.Read())
                {
                    entity.DocId = id;
                    entity.DocNumber = Int32.Parse(r["DocNum"].ToString());
                    entity.VersionNum = Int32.Parse(r["VersionNum"].ToString());
                    //etc., etc., etc.
                    entity.SecurityDescriptor = security.getSecurityDescriptor(r["DefaultSecurity"].ToString(), r["DocNum"].ToString(), r["VersionNum"].ToString());
                }
            }

            logger.writeItem(id);
            command.Dispose();
            connection.Close();
            connection.Dispose();
            return entity;            
    }

    public IEnumerable<DMSFile> ReadList(ref string BatchingCookie, ref System.String HasMoreActivities)
    {
            HasMoreActivities = "false";

            batchSize = logger.getBatchCount();

            SqlConnection connection = new SqlConnection(connectionString);
            SqlCommand command = new SqlCommand(cmd, connection);

            List<DMSFile> entityList = new List<DMSFile>(batchSize);
            if (entityList.Count > 0){entityList.Clear();}

            connection.Open();
            using (SqlDataReader rdr = command.ExecuteReader())
            {
                while (rdr.Read()) {
                    entityList.Add(new DMSFile() { 
                        DocId = rdr["DocNum"].ToString() + "." + rdr["VersionNum"].ToString(),
                        LastModified = DateTime.Parse(rdr["LastEdited"].ToString()).ToUniversalTime()
                    });
                }
            }


            if (entityList.Count == batchSize){
                HasMoreActivities = "true";
                BatchingCookie = entityList[entityList.Count - 1].DocId;
            }

            logger.writeAll(entityList.Count.ToString(), BatchingCookie);

            DMSFile[] entityArray = entityList.ToArray();

            command.Dispose();
            connection.Close();
            connection.Dispose();

            return entityArray;
    }

    public Stream ReadItemStream(string id)
    {
        SqlConnection connection = new SqlConnection(connectionString);
        SqlCommand command = new SqlCommand(cmd, connection);

        connection.Open();
        byte[] bytes = null;
            //SQL only return document location
        string fileLocation = command.ExecuteScalar().ToString();
        command.Dispose();
        connection.Close();
        connection.Dispose();


        try
        {
            using (FileStream fs = new FileStream(fileLocation, FileMode.Open, FileAccess.Read))
            {
                int length = Int32.Parse(fs.Length.ToString());
                bytes = new byte[length];
                fs.Read(bytes, 0, length);
            }
        }
        catch (Exception fileError){logger.logError(fileError.Message + "     " + fileLocation2);}

        return new MemoryStream(bytes);
    }
  • Have you profiled your connector code? Could the "leak" be in your code? Remeber that .net don't release memory unless a variable is not referenced any more and it won't garbage collect unless needed either. – Mikael Svenson Apr 17 '14 at 6:14
  • @Mikael-SharePointMVP I added some code to the post. This is the 4th or 5th version of the connector, each one returning less and less. If there's a glaring detail I am missing, I would love to know. I think I am disposing of all superfluous data, yet only small steps in memory management have been obtained. That said, the full crawl completes every time with (presumably) adequate efficiency. Thus, further complicating the issue. (Note* the "custom logging" was only recently added. This problem has existed long before it). – BJenkins Apr 17 '14 at 13:40
  • @Mikael-SharePointMVP Sorry, I didn't reply to your initial question. Yes, I have analyzed the code, which does highlight the need to dispose the sql connections after closing (a silly mistake). However, this still does not explain why I would only be experiencing issues during the incremental crawl (seemingly, a less intensive process). – BJenkins Apr 17 '14 at 14:51
  • Don't see any obvious errors except I would not use an SqlDataAdapter, but rather a SqlDataReader so as not to load more data into memory than needed at any given time. Also you can initialize the List<DMSFile>(batchSize) to avoid resizing it as grows. Initial size is 4 if I'm not mistaken. You might also be able to wrap the return out from ReadList with a yield return to optimize it even more. – Mikael Svenson Apr 22 '14 at 6:21
  • @Mikael-SharePointMVP . I made the proposed changes, but it seems to only have a negligible effect on memory consumption. I am still struggling to complete incremental crawls, which spend hours on the gathering process. ReadList is only being called every 30-60 minutes when batching 1 million items per call. This is occurring even when there are no updates to process (no ReadItem). The sql cmd only takes a few seconds to return records. Also, the gatherer is storing several GBs of information in the temp folder. Is this normal for incrementals? Once again, works great on smaller repository. – BJenkins Apr 30 '14 at 15:23
0

After a lot of fruitless research, I stumbled upon this KB article describing a very similar issue with the FAST search solution:

http://support.microsoft.com/kb/2601211

It would seem that the crawl box was gathering a lot more information than necessary, creating huge entities. After installing the hotfix: Memory usage decreased significantly, the 'Unable to Access Object' errors disappeared, and the crawl was able to complete.

We started experiencing issues with the next incremental crawl, but I believe those are environmental and unrelated to the original issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.