We migrated a SharePoint 2010 with about 30 site collections, each in it's own content database, to a newly setup SharePoint 2019. We played around with some sets of content sources to only index old content once, but ended up on using the default content source with the root site collection as single start address. The first full crawl indexed all 9.5 million documents fine (with a normal amount of warnings and errors...) in about 45 hours. Our goal was to start crawling incrementally from there on, to only index the new items. But the SharePoint server indexes all items again with each incremental crawl! So each incremental crawl is taking around 45 hours! There are definitely no changes to the old files as they are part of an archive. So no changes to permissions, metadata or content in those files occurs!

Can anybody shed some light on why the incremental crawl may need to index every file again? We did not find anything specific inside the ULS log or elsewhere. New files are added to the index, so it kind of does it's job...

EDIT: We have a new picture: The last crawl's duration was "only" 30h and only 6 of the 9 million documents were crawled. I'm trying another crawl to see, if it reduces the duration again.

  • My advice would be to Enable continuous crawls. And also, split content source into multiple when there are millions of items in the farm. Commented Feb 9, 2021 at 18:22
  • Hi. I considered this, but refrained from it due to the performance issues this could cause. I understood that the continuous crawl would start an incremental every 4 hours to address faulty documents. So this would leave me with continuous AND incremental crawls running in parallel. Also: edit in Post!
    – Urs
    Commented Feb 10, 2021 at 15:42
  • 1
    In our prod SharePoint 2013 environment, where we have 70 million plus documents, we use continuous crawl without any issue. In fact we set it up 3-minute refresh time. We have split content source at the site collection level so that, if needed, we can run full-crawl on a particular site collection independently. Our craw history logs show, continuous crawl handle 99% of items so no stretch to the incremental crawl Commented Feb 10, 2021 at 18:39
  • Could you please post a screenshot of Crawl Log -> Crawl History for further troubleshooting? Thanks!
    – MHeld
    Commented Feb 22, 2021 at 15:46
  • 1
    @MHeld: Sorry for the late response. Crawl duration actually came down to absolutely normal values (3-4 minutes for up to 1.400 documents in a 1 hour interval). I cannot see anything, that was the cause to this behavior, but actually it is gone.
    – Urs
    Commented Mar 9, 2021 at 7:35


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.