I'm looking for suggestions on how you would approach validating metadata on millions of documents that are spread across 10+ site collections.

All of the documents are tagged with metadata.
On-prem farm.
The system of record is a SQL table.
Currently PowerShell is used to set/update metadata on documents.

I can reuse some of the logic from the existing PowerShell set/update script but looping through ~100k SQL rows one at a time sounds really slow.

Would SharePoint batch processing + some sort of multi threading be a good idea?
When processing this much data, would there be any advantage to creating a .Net solution Vs. PowerShell?

1 Answer 1


Multithreding is definitely an option, You could have separate script for each of your site (done that myself on many migration projects), and you could execute it simultaneously. If you want to use batch processing then I would go with .net solution but that is only personal preference of working with web services.

  • Chunking out the SQL table into several different scripts or threads is likely the answer. i.e. each process will handle a couple thousand rows at a time. a temp table could be used to track progress so a script/thread doesn't cross process data. ... Thank you for the input. Jun 4, 2019 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.