0

We migrated a SharePoint 2010 with about 30 site collections, each in it's own content database, to a newly setup SharePoint 2019. We played around with some sets of content sources to only index old content once, but ended up on using the default content source with the root site collection as single start address. The first full crawl indexed all 9.5 million documents fine (with a normal amount of warnings and errors...) in about 45 hours. Our goal was to start crawling incrementally from there on, to only index the new items. But the SharePoint server indexes all items again with each incremental crawl! So each incremental crawl is taking around 45 hours! There are definitely no changes to the old files as they are part of an archive. So no changes to permissions, metadata or content in those files occurs!

Can anybody shed some light on why the incremental crawl may need to index every file again? We did not find anything specific inside the ULS log or elsewhere. New files are added to the index, so it kind of does it's job...

EDIT: We have a new picture: The last crawl's duration was "only" 30h and only 6 of the 9 million documents were crawled. I'm trying another crawl to see, if it reduces the duration again.

Improve this question
5
  • My advice would be to Enable continuous crawls. And also, split content source into multiple when there are millions of items in the farm.
    – Matiur Rahman
    Feb 9, 2021 at 18:22
  • Hi. I considered this, but refrained from it due to the performance issues this could cause. I understood that the continuous crawl would start an incremental every 4 hours to address faulty documents. So this would leave me with continuous AND incremental crawls running in parallel. Also: edit in Post!
    – Urs
    Feb 10, 2021 at 15:42
  • 1
    In our prod SharePoint 2013 environment, where we have 70 million plus documents, we use continuous crawl without any issue. In fact we set it up 3-minute refresh time. We have split content source at the site collection level so that, if needed, we can run full-crawl on a particular site collection independently. Our craw history logs show, continuous crawl handle 99% of items so no stretch to the incremental crawl
    – Matiur Rahman
    Feb 10, 2021 at 18:39
  • Could you please post a screenshot of Crawl Log -> Crawl History for further troubleshooting? Thanks!
    – MHeld
    Feb 22, 2021 at 15:46
  • 1
    @MHeld: Sorry for the late response. Crawl duration actually came down to absolutely normal values (3-4 minutes for up to 1.400 documents in a 1 hour interval). I cannot see anything, that was the cause to this behavior, but actually it is gone.
    – Urs
    Mar 9, 2021 at 7:35

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.