7

I have a list of SharePoint list item URLs and list item attachment urls. I need to download all the files from each one.

Overall:

  1. I have over 50 000 000 files to download over 200 farms.
  2. These files are spread over SharePoint environments that cover SharePoint 2010 to SharePoint 2019. There's also a few SharePoint Online versions to take into account.
  3. We're skipping any file over 80 MB in size. All files to not download have been skipped.
  4. Currently using a multi-threaded HTTP client program to download them. Each file is downloaded with a single HTTP request.

Is there some faster way to download all these files using a batch operation? For example, some way to get several files at once from a single web request instead of one-at-a-time?

Is there any way we can use a direct connection to the database to bulk download content?

Or is there a way to use the export feature to somehow accomplish this?

I'm open to suggestions, this process has been taking a very long time to complete...

Some reading I've done:

I read this article and it seems to say something about Blob to document conversion is the reason why this download process is slow. But I'm not positive.

I was doing some more reading at Crawling BLOBs in SharePoint.

Improve this question
10
  • Hey,This does seem viable to do via either batch or powershell. But I have a couple questions that might help: are these all coming from the same list? Or different lists? What format are you receiving the URLs? Is this modifiable?
    – KGlasier
    Commented Dec 13, 2018 at 21:56
  • its basically a dump of an entire sharepoint web application. so it will be, in order, all files from all lists from all sites from all site collections. I am getting the URLs from CSOM. all the powershell stuff I've seen so far is just a loop of http client requests like i am already doing.
    – Nicholas DiPiazza
    Commented Dec 13, 2018 at 22:35
  • 2
    Which version of SharePoint are you using? Approximately how many items are you trying to download and what is the approximate total size of them?
    – Damjan Tomic
    Commented Jan 10, 2019 at 17:17
  • 2
    I was doing some thinking and was wondering... why do you need to download everything? How far back does the data go? Have you considered the possibility of different data retention policies so you don’t have 50 million files? At that volume you might already be doing the fastest reasonable solution. You just have a lot of data.
    – KGlasier
    Commented Jan 12, 2019 at 19:20
  • 1
    I dumped all the info from these comments into the question.
    – Nicholas DiPiazza
    Commented Jan 12, 2019 at 20:38

1 Answer 1

Reset to default
1
+50

Generally a single http request if for accessing a single resource from the server. Given that I would do below

Example: 20 farms

  1. Segregate your farms into 20 different CSV (which would have the links). Each CSV would only have links for 1 unique farm
  2. Have 20 unique application instances (your custom coded app which downloads the files)

As others said, the blob conversion would surely take time. If your SharePoint environments already have SQL configured with Remote BLOB, then this would not have been the case as the download would have been more like FTP based.

Improve this answer
2
  • i am already doing this. sorry if my question wasn't clear. i am already to resorting to downloading each file one by one, and spreading them out to the farm. looking for better solutions. at the very least to reduce some of the http request overhead... or something
    – Nicholas DiPiazza
    Commented Jan 16, 2019 at 17:22
  • As I said, check if RBLOB is enabled and go that way. Provided you have to map the UNC path to the related SharePoint SQL table
    – Ananda Prasad Bandaru
    Commented Feb 1, 2019 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.