1

I'm looking for suggestions on how you would approach validating metadata on millions of documents that are spread across 10+ site collections.

All of the documents are tagged with metadata.
On-prem farm.
The system of record is a SQL table.
Currently PowerShell is used to set/update metadata on documents.

I can reuse some of the logic from the existing PowerShell set/update script but looping through ~100k SQL rows one at a time sounds really slow.

Would SharePoint batch processing + some sort of multi threading be a good idea?
When processing this much data, would there be any advantage to creating a .Net solution Vs. PowerShell?

0

Multithreding is definitely an option, You could have separate script for each of your site (done that myself on many migration projects), and you could execute it simultaneously. If you want to use batch processing then I would go with .net solution but that is only personal preference of working with web services.

  • Chunking out the SQL table into several different scripts or threads is likely the answer. i.e. each process will handle a couple thousand rows at a time. a temp table could be used to track progress so a script/thread doesn't cross process data. ... Thank you for the input. – Taco_Buffet Jun 4 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.