I recently performed an index reset, and did a full crawl on my 2 content sources. Afterwards I wanted to go enable Continuous Crawl on the content sources, but whenever I visit the content sources page it times out.
Message received on webpage:
Sorry, something went wrong
System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
ULS Logs from WFE01:
OWSTIMER.EXE (0x1BD0) 0x1920 SharePoint Foundation Monitoring b4ly Medium Leaving Monitored Scope (Timer Job job-diagnostics-performance-metric-provider). Execution Time=92.2191926627546 ee5e819c-6a60-a03b-8911-a03a0c19adab
OWSTIMER.EXE (0x1BD0) 0x1130 SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (ExecuteWcfOperation:http://tempuri.org/ISearchAdminServiceApplication/GetSharePointContentSources). Execution Time=120021.762332922 d15e819c-4a22-a03b-8911-a5a9c42df0c5
OWSTIMER.EXE (0x1BD0) 0x1130 SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (GetSharePointContentSources). Execution Time=120021.860320236 d15e819c-4a22-a03b-8911-a5a9c42df0c5
OWSTIMER.EXE (0x1BD0) 0x1130 SharePoint Server Search Query dka3 High SearchServiceApplicationProxy::GetSharePointContentSources--Error occured: System.TimeoutException: The request channel timed out while waiting for a reply after 00:02:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://app01:32843/cc9da127b0fd4fc9b5e916ebbb6c47fd/SearchService.svc' has exceeded the allotted timeout of 00:02:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) --- End of inner exception stack trace --- at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason) at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) --- End of inner exception stack trace --- Server stack trace: at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.Office.Server.Search.Query.ISearchAdminServiceApplication.GetSharePointContentSources() at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName) d15e819c-4a22-a03b-8911-a5a9c42df0c5
OWSTIMER.EXE (0x1BD0) 0x1130 Web Content Management Publishing agdjw High GetChanges :: Unable to get GetCrawledUrls for webapp db id = dc43530e-0a58-4f87-8823-90abdb249da2, query string = , start time = 03/27/2014 13:30:00 (UTC), end time = 03/27/2014 13:35:00 (UTC). Exception : System.TimeoutException: The request channel timed out while waiting for a reply after 00:02:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://app01:32843/cc9da127b0fd4fc9b5e916ebbb6c47fd/SearchService.svc' has exceeded the allotted timeout of 00:02:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) --- End of inner exception stack trace --- at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason) at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) --- End of inner exception stack trace --- Server stack trace: at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.Office.Server.Search.Query.ISearchAdminServiceApplication.GetSharePointContentSources() at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName) at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, WebServiceBackedOperation`1 webServiceCall) at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, WebServiceBackedOperation`1 webServiceCall) at Microsoft.SharePoint.Publishing.SearchChangeLogGeneratorJobDefinition.SearchChangeLog.GetChanges(DateTime currentSnapShotTime) d15e819c-4a22-a03b-8911-a5a9c42df0c5
OWSTIMER.EXE (0x1BD0) 0x1130 Web Content Management Publishing ajes2 Medium SearchChangeLogGeneratorJobDefinition.Execute(): Skipping central admin web application. d15e819c-4a22-a03b-8911-a5a9c42df0c5
OWSTIMER.EXE (0x1BD0) 0x1130 Web Content Management Publishing ajes4 Medium No changes were detected at this time 03/27/2014 13:35:00 d15e819c-4a22-a03b-8911-a5a9c42df0c5
OWSTIMER.EXE (0x1BD0) 0x1130 Web Content Management Publishing agdj3 Medium setSavedSearchCrawlTime: Saving the new search crawl time = '03/27/2014 13:35:00' d15e819c-4a22-a03b-8911-a5a9c42df0c5
OWSTIMER.EXE (0x1BD0) 0x1130 SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (Timer Job SearchChangeLogGeneratorJobDefinition). Execution Time=120037.747954 d15e819c-4a22-a03b-8911-a5a9c42df0c5
w3wp.exe (0x07CC) 0x056C SharePoint Server Search Query dka1 High SearchServiceApplicationProxy::GetUpdatedQueryParameterSpecifications--Proxy Name: Search Service Application Proxy EndPoint: http://app02:32843/cc9da127b0fd4fc9b5e916ebbb6c47fd/SearchService.svc 525d819c-aad5-a03b-8911-abca5e51671b
w3wp.exe (0x07CC) 0x056C SharePoint Server Search Query dk8z High SearchServiceApplicationProxy::GetChannel--Channel Creation time: 0 525d819c-aad5-a03b-8911-abca5e51671b
w3wp.exe (0x07CC) 0x056C SharePoint Foundation Topology e5mc Medium WcfSendRequest: RemoteAddress: 'http://app02:32843/cc9da127b0fd4fc9b5e916ebbb6c47fd/SearchService.svc' Channel: 'Microsoft.Office.Server.Search.Administration.ISearchServiceApplication' Action: 'http://tempuri.org/IQueryRuleOperations/GetUpdatedQueryParameterSpecifications' MessageId: 'urn:uuid:06005fe4-a5ae-4ac7-80a9-6219efb10fe0' 525d819c-aad5-a03b-8911-abca5e51671b
w3wp.exe (0x07CC) 0x056C SharePoint Server Search Query dka2 High SearchServiceApplicationProxy::GetUpdatedQueryParameterSpecifications--Id: Elapsed Time: 15.6 Proxy Name/ID: Search Service Application Proxy/e2e5ceda-d237-48fb-a44c-84dc408fe303 EndPoint: http://app02:32843/cc9da127b0fd4fc9b5e916ebbb6c47fd/SearchService.svc 525d819c-aad5-a03b-8911-abca5e51671b
w3wp.exe (0x07CC) 0x17A0 SharePoint Portal Server Runtime 8gp7 Medium Topology cache updated. (AppDomain: /LM/W3SVC/1509824234/ROOT-1-130403733001033259) 545d819c-4ab8-a03b-8911-af6757732669
OWSTIMER.EXE (0x1BD0) 0x19EC SharePoint Server Search Query dka3 High SearchServiceApplicationProxy::GetSharePointContentSources--Error occured: System.TimeoutException: The request channel timed out while waiting for a reply after 00:02:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://app01:32843/cc9da127b0fd4fc9b5e916ebbb6c47fd/SearchService.svc' has exceeded the allotted timeout of 00:02:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) --- End of inner exception stack trace --- at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason) at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) --- End of inner exception stack trace --- Server stack trace: at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.Office.Server.Search.Query.ISearchAdminServiceApplication.GetSharePointContentSources() at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName) 1a5f819c-ea58-a03b-8911-a75ad6fbdb6e
OWSTIMER.EXE (0x1BD0) 0x19EC SharePoint Foundation Topology 8313 Critical A failure was reported when trying to invoke a service application: EndpointFailure Process Name: OWSTIMER Process ID: 7120 AppDomain Name: DefaultDomain AppDomain ID: 1 Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:cc9da127b0fd4fc9b5e916ebbb6c47fd#authority=urn:uuid:ee79b310097b4c1f942abd42f1246843&authority=https://app01:32844/Topology/topology.svc Active Endpoints: 1 Failed Endpoints:1 Affected Endpoint: http://app01:32843/cc9da127b0fd4fc9b5e916ebbb6c47fd/SearchService.svc 1a5f819c-ea58-a03b-8911-a75ad6fbdb6e
ULS Logs from APP01:
OWSTIMER.EXE (0x0DB8) 0x2E8C SharePoint Foundation Configuration a16e High SPAce PrincipalName found account renamed to NULL SID. Using new name. 76198530-481a-4bd9-8da7-cbd9c4ea9861
w3wp.exe (0x1E78) 0x2750 SharePoint Foundation General 8nca Medium Application error when access /_admin/search/listcontentsources.aspx, Error=System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at Microsoft.Office.Server.Search.Internal.UI.SearchCentralAdminPageBase.ErrorHandler(Object sender, EventArgs e) at Microsoft.Office.Server.Search.Internal.UI.SearchCentralAdminPageBase.OnError(EventArgs e) at System.Web.UI.Page.HandleError(Exception e) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest() at System.Web.UI.Page.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) 005f819c-face-a03b-d6bb-e91a74abebe0
w3wp.exe (0x1E78) 0x2750 SharePoint Foundation Runtime tkau Unexpected Microsoft.SharePoint.SPException: System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at Microsoft.Office.Server.Search.Internal.UI.SearchCentralAdminPageBase.ErrorHandler(Object sender, EventArgs e) at Microsoft.Office.Server.Search.Internal.UI.SearchCentralAdminPageBase.OnError(EventArgs e) at System.Web.UI.Page.HandleError(Exception e) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest() at System.Web.UI.Page.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) 005f819c-face-a03b-d6bb-e91a74abebe0
w3wp.exe (0x1E78) 0x2750 SharePoint Foundation General ajlz0 High Getting Error Message for Exception Microsoft.SharePoint.SPException: System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at Microsoft.Office.Server.Search.Internal.UI.SearchCentralAdminPageBase.ErrorHandler(Object sender, EventArgs e) at Microsoft.Office.Server.Search.Internal.UI.SearchCentralAdminPageBase.OnError(EventArgs e) at System.Web.UI.Page.HandleError(Exception e) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest() at System.Web.UI.Page.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) 005f819c-face-a03b-d6bb-e91a74abebe0
w3wp.exe (0x1E78) 0x2750 SharePoint Foundation General aat87 Monitorable 005f819c-face-a03b-d6bb-e91a74abebe0
w3wp.exe (0x1E78) 0x2750 SharePoint Foundation Micro Trace uls4 Medium Micro Trace Tags: 0 nasq,5 agb9s,68 e5mc,29 e5mc,120020 8nca,0 tkau,0 ajlz0,1 aat87 005f819c-face-a03b-d6bb-e91a74abebe0
w3wp.exe (0x1E78) 0x2750 SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (ExecuteWcfOperation:http://tempuri.org/ISearchApplicationAdminWebService/GetContentSources). Execution Time=120041.605776712 005f819c-face-a03b-d6bb-e91a74abebe0
ULS Logs on APP02 didn't seem to have anything relevant.
Based on some googling I thought it may be a large crawl log causing the timeout, so I tried setting the $searchApp.CrawlLogCleanUpIntervalInDays = 1 hoping it would cleanup some history and possibly fix the timeout. I did that 3 days ago, but the timeout is still happening.
