Once in a while my full-crawl fails and stops working properly. The crawllogs show this error for all items crawled.
The item could not be indexed successfully because the item failed in the indexing subsystem. ( The item could not be indexed successfully because the item failed in the indexing subsystem.; Caught exception when preparing generation GID[7570]: (Previous generation (last=GID[7569], curr=GID[7570]) is still active. Cannot open GID[7570]); Aborting insert of item in Link Database because it was not inserted to the Search Index.; ; SearchID = F201681E-AF1B-45D2-BFFD-6A2582D10C19 )
The full crawls starts out ok, after a while (1.5 hours into the process, 50% of all the data) suddenly no more items can be added to the index.
The index seems to be stuck. The index files on disk are no longer updated (Located in D:\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\BAADC4\IndexComponent3\storage\data\SP4d91e6081ac3.3.I.0.0\ms\%default)
The Index and Admin component start to report these error in the ULS logs:
NodeRunnerIndex: Journal[SP4d91e6081ac3]: Rolling back GID[7570] to GID[7569] prepGen=GID[7569]
NodeRunnerIndex: Remote service invocation: method=RollbackGeneration() Service = { Implementation type=Microsoft.Ceres.SearchCore.ContentTargets.IndexRouter.IndexRouter Component: SP4d91e6081ac3I.0.0.IndexRouter Exposer Name: GenerationContentTarget} terminated with exception: System.InvalidOperationException: Illegal state transition in SP4d91e6081ac3I.0.0.FastServer.FSIndex: Rollback -> Rollback
NodeRummerAdmin: RetryableInvocation[SP4d91e6081ac3]: Exception invoking index cell I.0.0. Retrying in 16 seconds: System.InvalidOperationException: Illegal state transition in SP4d91e6081ac3I.0.0.FastServer.FSIndex: Rollback -> Rollback
Looks to me the index has troubles updating/merging 'generations'. But the exact working of the indexer is not documented (as far is I know). Let alone how to fix this.
Other (maybe related) observations
Just before the errors start the NodeRunnerIndex starts a 'checkpoint': Journal[SP4d91e6081ac3]: Starting checkpoint because forceCheckpoint is true. which ends a few moments later with Journal[SP4d91e6081ac3]: All journal users have completed checkpoint Checkpoint[7560-7569].
Also just before the errors start to appear a TimerJob starts: Name=Timer Job job-application-server-admin-service. This timerjobs does some strange things to the search topology: Synchronizing Search Topology for application 'Search Service Application' with active topology [...] and Activating components. Previous topology: --- New Topology: TopologyId: [...] followed by Starting to execute Index RedistributeData method.
And right after these two evente the errors start to occur. (each row is a ULS log enrty)
INFO : fsplugin: IndexComponent3-bd83a8aa-923b-4526-97e8-47eac0986ff7-SP4d91e6081ac3.I.0.0 (4236): Prepare generation: 324 documents
IndexRouter[SP4d91e6081ac3]: Caught exception when preparing generation GID[7570]: (External component has thrown an exception.): System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
GenerationDispatcher[SP4d91e6081ac3]: Failed to prepare GID[7570] in 453 ms, failed on cells: [I.0.0], stale services: []
Journal[SP4d91e6081ac3]: Rolling back GID[7570] to GID[7569] prepGen=GID[7569]
Remote service invocation: method=RollbackGeneration() Service = { Implementation type=Microsoft.Ceres.SearchCore.ContentTargets.IndexRouter.IndexRouter Component: SP4d91e6081ac3I.0.0.IndexRouter Exposer Name: GenerationContentTarget} terminated with exception: System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
RetryableInvocation[SP4d91e6081ac3]: Exception invoking index cell I.0.0. Retrying in 2 seconds: System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
Journal[SP4d91e6081ac3]: Rolling back GID[7570] to GID[7569] prepGen=GID[7569]
The Question What is causing this? And how to prevent it? It happend twice in two weeks nows. Out of the blue, no config change has been made, all disks have enough space.
Known Fix This resolves the problem, but doesn's address the root-cause of the problem!
- Stop all crawls.
- Wait a few minutes to let the crawl come to a complete stop.
- Reset the index (clearing all!)
- Start a full-crawl. In the meantime no search is available to the end user (boohoo!)
