2

Farm: SharePoint Server 2010

This farm uses Remote Blob Storage (Storage Maximizer by Dell)

I'm doing a full crawl of a content source with approximately 1.4 million items in it. All items are marked as successfuly crawled for the first 650,000 or so items. After that, I start to see event id 114 in the Windows event log indicating the Search Gatherer cannot reach the server. Once that appears, every item is marked as an error when it is being crawled.

The exact message:

"The server xxxxxxxxxx is unavailable and could not be accessed. The server is probably disconnected from the network. Items from this server will be skipped by the crawl for the next 10 minutes"

I can resolve to that URL from a browser session on the crawl server where the error is being thrown. I say this to state that the URL is in fact available from that server. Note: the hosts file on this crawl server is pointing to itself for this URL.

I've reset the index and initiated a full crawl 3 separate times. Each time this error starts after its successfully crawled over a half million items. Thinking this might be resource-related, but not sure what to look at.

I have a Windows Server admin watching disk, memory, CPU but he doesn't see anything abnormal.

Any ideas what could be causing this issue?

  • Is it always failing on a particular site collection? Anything special about the site where it fails? Large files? – Eric Alexander Aug 1 '16 at 16:33
  • It appears to be indiscriminate. Throwing an error for any site collection once that error starts appearing in the event log. Errors in the magnitude of >700,000 items. – Joanne Klein Aug 1 '16 at 16:47
  • Have you checked the Web Front Ends during the process? Also take a look at blendmaster.net/blog/2012/09/… – Amal Hashim Aug 1 '16 at 17:08
  • It's always permissions... Log in to the server hosts your search service application as the Search Crawler account. Open IE to the failing URL. Make sure the service account has read only. All else fails, rebuild index. – iOnline247 Aug 1 '16 at 19:38
  • The Search content access account has access. Confirmed by successfully browsing to the failed URL with the content access account. When you say 'rebuild index', you mean 'reset index'? I've done that a few times already and restarted the full crawl. After several hours of the crawl running it starts to error out. I'm on a support call with Microsoft right now - they're thinking it might be network saturation so we increased the timeout from 60,60 to 150,150 and retrying full crawl. – Joanne Klein Aug 1 '16 at 19:43
1

This farm uses 3rd party Remote Blob Storage (Dell's Storage Maximizer) and after opening up a Premier Support Incident with MSFT it was determined that eventually the remote blob store was getting queued up with all the requests and was exceeding the default of 60 seconds and timing out. By increasing the timeout setting in Search Administration from the default 60 seconds to 150 seconds, this was enough for the blob to be returned before SharePoint would send another request. This prevented the requests from getting backed up. Without this in place, the requests would get queued up and no search requests were being fulfilled.

At a high level, Microsoft described the root cause of this problem as "network congestion". There were no CPU, disk or memory constraints on the SharePoint server. The issues were focused on the time it took to retrieve the blob from the file system on a large scale. (i.e. the problem would only manifest itself after a half million blobs or so)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.