5

I have an external partner account that wants to download a large GIS data set from our SharePoint. The data set is about 13500 files and is around 43 gb in size, and is housed within a folder. When they try to download it, it's downloaded as a zip file, but it's never able to fully complete, even though the browser says the download is complete. Sometimes the zip file is only a couple of GBs and won't open.

enter image description here

Other times, the download manages to get to around 20 gb, and the zip file can be opened, but half the files haven't been downloaded and I get a bunch of text files in the zip with error messages in them.

enter image description here

Each text file has an error message inside: "This file cannot be downloaded.

ExceptionType: ZipMeTAException.

CorrelationId: 7dcb10e9-be4f-4edb-b24b-d469dcaaf844,

UTC DateTime: 1/29/2020 10:09:47 PM"

I've tried downloading the folder myself and never got past 20 gb before the download "completed". Is there any reason why it's not possible to download the whole folder, and is there a better method to download such a huge number of files that would work for an external partner?

Improve this question

4 Answers 4

Reset to default
1

If the folder contains 13500 files then "Open with Explorer" will not work because you exceed the List View Threshold LVT (default 5000). Increasing the LVT is not possible for SharePoint Online. Alternatives:

  1. Use PowerShell (search for scripts that provide this capability). Check upfront if the scripts can handle large lists.
  2. Use tools that can download large number of items and that support LVT. e.g. here
  3. Use OneDrive for Business to sync the files

Paul

Improve this answer
1
  • Regarding 3rd option 'Use OneDrive for Business to sync the files', is it possible to sync document library to local machine AND THEN COPY the 'local version' of the files to another destination on local machine eg 'C:\Users\Me\Downloads\My Copied Files' so that I can rename copied files and upload them to another document library? I tried to download the folder (Step 1) and it would only download 20GB of a 50GB folder, so syncing seems like my only option. I have the 'Files On Demand' feature turned on in OneDrive, so not sure if I can copy synced folders/files like that.
    – user1063287
    Sep 21, 2021 at 7:39
0

You can return to Classic view in the library, and try to use "Open with Explorer" feature to open the folder, then copy and paste the files into your local. enter image description here

Have a try and let's know the result.

Improve this answer
1
  • 1
    I tried that, but when opening a folder that should have about 13500 files inside, the folder instead was empty of any files, even with reloading the folder.
    – gtew234
    Jan 31, 2020 at 23:36
0

I had similar problem. I had to use Explorer to open OneDrive folder then copy the files in batches. Like 10-20 files at a time that totaled about 1-2 GBs. If Explorer list view creates a limitation, I would suggest viewing it a command prompt and manipulating there or using PowerShell as mentioned earlier. I know this is late, however someone else may run into this issue. It was a pain, as it took all day, but it was finally done.

Improve this answer
0

I've had the same problem. And yes, it's a real pain because:

  1. It takes ages to download large data sets
  2. There seems to be no rhyme nor reason to it.

Sometimes I can download folders as large as 500 MB without any hassle, other times I have to open up folders and download individual files of 30 MB each.

I haven't experienced this problem with other cloud sharing platforms such as Google Drive and Dropbox. The problem must therefore lie buried somewhere within SharePoint.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.