We're facing slow crawl times on our main content source and I'm not sure what the cause is.
The farm (SP2010) consists of 2 WFE's and 3 APP servers. On all APP servers, crawl components have been deployed. The index / query components are located on the WFE's. Any comments on the architecture are welcome. The content source crawls two SharePoint web applications which together contain about a million items.
Now what I'm seeing is that the initial performance is fine. In the first few hours, almost everything is crawled and propagated into the index. But then everything seems to slow down to almost a halt (0.05 items a second). Eventually everything finishes and the incrementals begin. These run very irregular, some times it finishes in 15 minutes and other times it would take several hours to finish.
Query performance is ok and my users can find their stuff, but because of the running full crawl, incrementals are not started. Due to migrations the content was pretty dynamic some months ago, but it's stable now.
So my question: what could be causing this lag, or what's a good way to find out?
Farm is patched untill december 2013 patches. Search filter pack has been updated to the latest version as well.
Hardware specs: 2 WFE's which both host a query component. 3 APP's which all host a crawl component. Index is spread over all servers.
All servers are equal in specs: 12GB RAM, Xeon quad core @ 2GHz.
Update April 4th: I splitted up the content source in order to be able to narrow down the problem. I found out that the culprit is our most locked down web application. The weird thing is that on this web application, we are enforcing the usage of AD groups, so there are almost no individual accounts in place other then some administrative ones. According to what I read on security only crawls, those would be prevented by using AD groups. So that's weird and leaves me to think the security only crawl is not the culprit here.
What I DO see, is quite a few filter related errors:
- "The filter daemon did not respond within the timeout limit."
- "The filtering process could not be initialized when trying to process this item. Verify that the file extension is a known type and the item is not corrupt."
- "The item may be too large or corrupt. You may also verify that you have the latest version of this IFilter"
Could have something to do with it, although the sum of these is only about 100 errors. I wouldn't expect 100 timeouts to be the cause of days of delay.