We have a test environment that is no longer able to successfully crawl content. The crawl logs show successful URLs, but I can't search for that content. It has worked in the past, but stopped working for some unknown reason.
This is a single SharePoint 2013 server that is crawling SharePoint 2010 farms. When looking at the search topology in the Search Service Application, I see:
I can also look at it with PowerShell and see that it is "degraded":
I tried resetting the index, but that did not help. Looking at the ULS log, I see lots of errors. Here are 3 examples:
CSSFeedersManager::session_CallbackReceived: Document 55747 processing failed. Error messages: Failed to create session with indexer; Aborting insert of item in Link Database because it was not inserted to the Search Index.; , errorID = 2147749241 hostName = EATIFSPS2013A nodeName = ContentProcessingComponent1
Component and System=Content1-a9f229cd-e4ce-43b6-88c3-fab1cc9b6d45, Correlation ID=d57ff765-1ffc-4fc9-bb65-7e0f11e9abfb, Tenant ID=0c37852b-34d0-418e-91c6-2ac25af4be5b Error code=800000, Flow Name=Microsoft.CrawlerFlow, Operator Name=InjectedOpForExceptionHandling/EventHybrid_Indexing, Message=The processing of item fails with error Failed to create session with indexer
Microsoft.Ceres.ContentEngine.Processing.Mars.MarsWriterProducer: Failed to create session for index system: SPb0bb381ca970: Microsoft.Ceres.SearchCore.Services.ContentRouter.ContentException: Unable to connect to index system SPb0bb381ca970
at Microsoft.Ceres.SearchCore.ContentRouter.ContentRouter.VerifyIndexSystem(String indexSystem)
at Microsoft.Ceres.SearchCore.ContentRouter.ContentRouter.OpenSession(String indexSystem, Action
1 callback, IDictionary2 defaultPolicy)
at Microsoft.Ceres.ContentEngine.Processing.Mars.MarsWriterProducer.GetSession(String indexSystem)
I have seen where I you have multiple index partitions you can remove one and add another. If you only have one, I have read that you have to create a new search service application (as you cannot remove all index partitions). Maybe I could add one and then remove the original/bad one.
I have also seen on this blog where maybe I should open up TCP 808 ports inbound into my SharePoint 2010 sites (the content I am crawling). We didn't close these on purpose, but maybe that did happen (I'm looking into it).
EDIT: I tried resetting the index and only crawling content on the single SharePoint 2013 machine and still have the problem. I don't think it has anything to do with firewall ports (TCP 808 or other).