Issue: When performing any type of search whether that be from the Enterprise Search Center or from Query Rules to test a query, the page results in an error.
Error on Manage Query Rules and Enterprise Search pages: Search has encountered a problem that prevents results from being returned.
Farm background: New environment being stood up. All servers are running Windows Server 2012 R2 with 24GB RAM The database servers 2 running MS SQL Server 2012 R2 with 32 GB ram We are using windows authentication as well as ADFS. All user UPN's have been updated in Active Directory to the domain UPN. SSL is implemented using standard port 443.
Topology: 2 WFEs --Search Query Processing Component --Search Admin Component 3 Application Servers --Search Crawler Component --Search Content Processing Component --Search Analytics Processing --Search Index Partition 2 DBs
Non-SP Servers: 1 Office Web Apps 1 ADFS 1 Workflows 1 proxy
Search does crawl successfully and there are approximately 400k items in the index. Seems to be related only to the Query Process.
ULS log shows: Name=Request (POST:https://thehive.xxxx.org:443/search/_vti_bin/client.svc/ProcessQuery) b613bb9c-fc4a-a020-6d41-0a96138145fb
Non-OAuth request. IsAuthenticated=True, UserIdentityName=0e.t|adfs provider|[email protected], ClaimsCount=11 b613bb9c-fc4a-a020-6d41-0a96138145fb
Name=Timer Job SchedulingUnpublish b613bb9c-3c4e-a020-6d41-01bfb551c3a0
SearchServiceApplicationProxy::GetQueryParameterSpecification--Proxy Name:Search Application Proxy EndPoint: http://webserver1:32843/04a32ef4861448aa873b85a2a67b1b20/SearchService.svc b613bb9c-fc4a-a020-6d41-0a96138145fb
SearchServiceApplicationProxy::GetChannel--Channel Creation time: 3.9996 b613bb9c-fc4a-a020-6d41-0a96138145fb
WcfSendRequest: RemoteAddress: 'http://web1:32843/04a32ef4861448aa873b85a2a67b1b20/SearchService.svc' Channel: 'Microsoft.Office.Server.Search.Administration.ISearchServiceApplication' Action: 'http://tempuri.org/IQueryRuleOperations/GetQueryParameterSpecification' MessageId: 'urn:uuid:2ffa95a4-0a71-4866-a3fb-2e33b6a0fe49' b613bb9c-fc4a-a020-6d41-0a96138145fb
Leaving Monitored Scope (Timer Job SchedulingUnpublish). Execution Time=4.72553075879756 b613bb9c-3c4e-a020-6d41-01bfb551c3a0
Name=Timer Job TranslationExport b613bb9c-bc4e-a020-6d41-04133823d269
TranslationExport has started b613bb9c-bc4e-a020-6d41-04133823d269 **Unable to write service call usage entry.** b613bb9c-fc4a-a020-6d41-0a96138145fb
SearchServiceApplicationProxy::GetQueryParameterSpecification--Error occured: System.ServiceModel.Security.**SecurityAccessDeniedException: Access is denied**. Server stack trace: at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter) at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Ch... b613bb9c-fc4a-a020-6d41-0a96138145fb
09/22/2014 13:30:04.01* w3wp.exe (0x2FEC) 0x2CE8 SharePoint Server Search Query dka5 High ...annels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.Office.Server.Search.Query.Rules.IQueryRuleOperations.GetQueryParameterSpecification(SearchObjectFilter filter) at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName) b613bb9c-fc4a-a020-6d41-0a96138145fb
QueryParameterCache::GetQueryParameters exception: System.ServiceModel.Security.SecurityAccessDeniedException: **Access is denied**. Server stack trace: at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter) at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.... b613bb9c-fc4a-a020-6d41-0a96138145fb
09/22/2014 13:30:04.01* w3wp.exe (0x2FEC) 0x2CE8 SharePoint Server Search Query ajh4h High ...Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.Office.Server.Search.Query.Rules.IQueryRuleOperations.GetQueryParameterSpecification(SearchObjectFilter filter) at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName) at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, Int32 timeoutInMilliseconds, Int32 wcfTime... b613bb9c-fc4a-a020-6d41-0a96138145fb
09/22/2014 13:30:04.01* w3wp.exe (0x2FEC) 0x2CE8 SharePoint Server Search Query ajh4h High ...outInMilliseconds, WebServiceBackedOperation`1 webServiceCall) at Microsoft.Office.Server.Search.Query.Pipeline.Utilities.QueryParameterCache.<>c__DisplayClass1.<GetQueryParameters>b__0() at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3() at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode) at Microsoft.Office.Server.Search.Query.Pipeline.Utilities.QueryParameterCache.GetQueryParameters(Guid tenantId, Guid farmId, Guid siteId, Guid webId) b613bb9c-fc4a-a020-6d41-0a96138145fb
I can manually access the endpoints in the browser. We have tried using the setup/admin account, farm account, local administrators and domain administrator accounts. Same results. I granted each of those same users access to the Search Service Application with Full Control. Have rebuilt the Search Service a dozen times or so. I've checked IIS to ensure that the endpoints were there on both WFE's and they were with security accounts provided access.