0

For the test purposes, we are trying to upload 1 million documents to document library in SharePoint Online. If everything goes really well, it will take approximately 7 days using our current script (see below).

Question: is there a bulk-upload technique we can use to generate a very large number of documents?

Current script that does not support bulk upload

# =================================================================== #
#              Script for Generating 1M Documents
# =================================================================== #
$SiteURL = "https://contoso.sharepoint.com/sites/ClassicTeam"
Connect-PnPOnline -Url $SiteURL -UseWebLogin

# =================================================================== #
#              Choose destination library
# =================================================================== #
$DocumentLibrary = 'Shared Documents'

# =================================================================== #
#               Upload 100K Documents
# =================================================================== #
$documentsToGenerate = 1000000

$files = Get-ChildItem -LiteralPath .\SampleFiles

$stopWatch = [system.diagnostics.stopwatch]::StartNew()
for($i=0;$i -lt $documentsToGenerate; $i++){
    $fileIndex = $i % $files.Count
    $localFile = $files[$fileIndex]
    $fineSuffix = (Get-Date).ToString("yyyyMMdd_hhmmss_ffff")
    $destinationFileName = $localFile.Name.Split(".")[0] +$fineSuffix + "." + $localFile.Name.Split(".")[1]
    $file = Add-PnPFile -Path $localFile.FullName -Folder $DocumentLibrary -NewFileName $destinationFileName
    Write-host [$i/$documentsToGenerate] " " -NoNewline -ForegroundColor Yellow
    Write-host  $file.Name "uploaded" -ForegroundColor Green 
}
$stopWatch.Stop()
Write-Host Time it took to create $documentsToGenerate documents in the $($SiteURL+$DestinationFolderUrl) -ForegroundColor Cyan
$stopWatch

1 Answer 1

4

Try this:

  • create a sample document; for instance demodoc.pdf in a folder called DemoFolder
  • use the below PowerShell script to multiply the document demodoc.pdf into one million copies, each with unique names:
$NumArray = (1..1000001)
ForEach ($number in $numArray) {
  Copy-Item "C:\Users\admin\Documents\DemoFolder\Demodoc.pdf" `
  -Destination "C:\Users\admin\Documents\DemoFolder\Demodoc$number.pdf"
}
  • Then use the free SharePoint migration tool to migrate the one Million documents from the local computer to a document library in SharePoint online.

Obviously, the speed of the migration will depend on a number of factors such as the size of the documents, your internet connection, the speed of the local computer, etc. I tried this in a virtual machine on Microsoft Azure and the entire process was completed in 22 hours.

enter image description here enter image description here

2
  • That's a very good option, thank you! I will accept it if there are no other suggestions within a week. And thank you for actually trying to upload a million documents. That's very nice of you! Commented Oct 11, 2020 at 15:31
  • 1
    Thanks for thanking me. The entire process of generating and uploading the documents to SharePoint online completed in approximately 22 hours.
    – Donspeck
    Commented Oct 12, 2020 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.