I have a list of SharePoint list item URLs and list item attachment urls. I need to download all the files from each one.
- I have over 50 000 000 files to download over 200 farms.
- These files are spread over SharePoint environments that cover SharePoint 2010 to SharePoint 2019. There's also a few SharePoint Online versions to take into account.
- We're skipping any file over 80 MB in size. All files to not download have been skipped.
- Currently using a multi-threaded HTTP client program to download them. Each file is downloaded with a single HTTP request.
Is there some faster way to download all these files using a batch operation? For example, some way to get several files at once from a single web request instead of one-at-a-time?
Is there any way we can use a direct connection to the database to bulk download content?
Or is there a way to use the export feature to somehow accomplish this?
I'm open to suggestions, this process has been taking a very long time to complete...
Some reading I've done:
I read this article and it seems to say something about Blob to document conversion is the reason why this download process is slow. But I'm not positive.
I was doing some more reading at Crawling BLOBs in SharePoint.