Moving large document libraries to new site collections

Question

I have a site collection with a dozen or so document libraries in it (total size about 300GB's). The goal is to move each of these document libraries into it's own site collection (and content database).

I've seen posts like Exporting sites to a new content database (talks about moving whole sites) and Restructuring a Very Large Document Library (restructuring a large document library). I found an article online that specifically adresses this issue, http://pravahaminfo.blogspot.com/2011/05/how-to-move-document-library-from-one.html, but the method of exporting the list as a template with content and then importing it on the new site only works up to 10 MB's (500 Mb's max if you change it) and my smallest document library is about 5GB's.

Right now I am considering exporting the list as a template WITHOUT content, importing it, and then using code to move the files. I looked into using Explorer view to move the files but the document libraries I have with over 20,000 folders in the root won't open in Explorer view...

Suggestions on how I should move forward? I'm currently looking for a method that will work in 2007 BUT we are migrating to 2010 in the near future so if the time/headache savings are big enough I will wait.

Answer 1

I am going to provide an answer that will work in 2010. I will need to depend on others who are more versed in MOSS for that; however, I believe there is a similar stsadm command for the Powershell I will provide below.

The Export-SPWeb command is able to be defined for a site, list or library. I have used this extensively to move a troubling list from production to dev/test so that I may work on a specific criteria.

Here is the TechNet Article

An example:

Add-PSSnapin Microsoft.SharePoint.PowerShell
$URL = "http://iwanttotest.com/MyManagedPath/SiteName"
$DocLibrary = "http://iwanttotest.com/apps/site/DocLibraryName"
Export-SPWeb -Identity $URL -Path C:\SiteBackup.cmp -ItemURL $DocLibrary -IncludeUserSecurity -IncludeVersions

Please note this must be executed on a server within the SP Farm so you may add the Powershell snapin.

Answer 2

Introduction

I had to move a document library which was bigger than the 524,288,000 byte maximum from an old SharePoint 2010 to SharePoint 2019. So, I came up with this solution. It's not perfect, but it works. Please don't run it blind as it could really mess up your server.

Caveats

• My source is a SharePoint 2010, my target is SharePoint 2019. This means that I ran the Source part of the script on PowerShell 2 and the Target part on PowerShell 5.
• I ran this solution from an account with full administrative credentials in SharePoint.
• My library had no root files, only folders.
• I was unable to run $Site=Get-SPSite "http://<sharepoint site>"; $Site.RecalculateStorageMetrics(), so I was unable to calculate any sizes. Therefore, I had to guess as to how many parts to break it into to get a size smaller than the maximum.
• The function Copy-SPFiles is a prerequisite for both the Source and the Target sections.

Overview

Source:

1. Create an empty template of the library I want to export
2. Create $Parts number of new libraries from the empty template
3. Determine the integer number of folders in each part, and save the remainder as $Adjust
4. Copy $Parts number of folders in sequence into each sub-library
5. Copy $Adjust number of folders in sequence into the last library
6. Save sub-libraries as Templates with content preserved
7. Delete sub-libraries
8. Export the Content Templates to a file
9. Delete the Content Templates

Target:

1. Determine the number of files to import
2. Import each file into the List Template Gallery
3. Create the libraries from the templates
4. Delete the Content Templates (Keep the empty one)
5. Copy the files from the sub-libraries into the target library
6. Delete the sub-libraries
7. Delete the files

Code

function Copy-SPFiles {
  [CmdletBinding(SupportsShouldProcess=$false)]
  Param ([Parameter(Mandatory=$true, ValueFromPipeline=$false)] [ValidateNotNullOrEmpty()] [Microsoft.SharePoint.SPFolder]$SourceFolder,
          [Parameter(Mandatory=$true, ValueFromPipeline=$false)] [ValidateNotNullOrEmpty()] [Microsoft.SharePoint.SPFolder]$TargetFolder,
          [Parameter(Mandatory=$false, ValueFromPipeline=$false)] [int]$MaxRootFoldersToCopy=[System.Int32]::MaxValue,
          [Parameter(Mandatory=$false, ValueFromPipeline=$false)] [switch]$BeyondRoot)
          #[Parameter(Mandatory=$false, ValueFromPipeline=$false, DontShow=$true)] [switch]$BeyondRoot)

  Begin { Write-Verbose "Entering $($PSCmdlet.MyInvocation.MyCommand.Name)" }

  Process {
    Write-Verbose "Processing $($SourceFolder.Files.Count.ToString()) files"
    for ($i=0; $i -lt $SourceFolder.Files.Count; $i++) {
      Write-Verbose "Processing File: $($SourceFolder.Files[$i].Name)"
      $NewFile = $TargetFolder.Files.Add($SourceFolder.Files[$i].Name, $SourceFolder.Files[$i].OpenBinary(),$True)

      # Copy Meta-Data from Source
      for ($j=0; $j -lt $SourceFolder.Files[$i].Item.Fields.Count; $j++) {
        if (!$SourceFolder.Files[$i].Item.Fields[$j].ReadOnlyField) {
          if ($NewFile.Item.Fields.ContainsField($SourceFolder.Files[$i].Item.Fields[$j].InternalName)) {
          $NewFile.Item[$SourceFolder.Files[$i].Item.Fields[$j].InternalName]=$SourceFolder.Files[$i].Item[$SourceFolder.Files[$i].Item.Fields[$j].InternalName]
          }
        }
      }
      $NewFile.Item.UpdateOverwriteVersion()
    }
 
    #Process SubFolders
    Write-Verbose "Processing $($SourceFolder.SubFolders.Count.ToString()) folders"
    if ($BeyondRoot.IsPresent) { $Limit=$SourceFolder.SubFolders.Count }
    else {
      if ($MaxRootFoldersToCopy -lt $SourceFolder.SubFolders.Count) { $Limit=$MaxRootFoldersToCopy } else { $Limit=$SourceFolder.SubFolders.Count }
      if ($Limit -gt $SourceFolder.SubFolders.Count) { $Limit=$SourceFolder.SubFolders.Count }
    }
    if (!$BeyondRoot.IsPresent) { Write-Verbose "But only up to $Limit" }
    for ($i=0; $i -lt $Limit; $i++) {
      Write-Verbose "Processing Folder: $($SourceFolder.SubFolders[$i].Name)"
      if ($SourceFolder.SubFolders[$i].Name -ne "Forms") {
        # Check if Sub-Folder exists in the Target Library
        $NewTargetFolder = $TargetFolder.ParentWeb.GetFolder($SourceFolder.SubFolders[$i].Name)
        if (!$NewTargetFolder.Exists) {
          $NewTargetFolder = $TargetFolder.SubFolders.Add($SourceFolder.SubFolders[$i].Name)
        }
        # Call the function recursively
        Copy-SPFiles -SourceFolder $SourceFolder.SubFolders[$i] -TargetFolder $NewTargetFolder -MaxRootFoldersToCopy $MaxRootFoldersToCopy -BeyondRoot
      }
    }
  }

  End { Write-Verbose "Exiting $($PSCmdlet.MyInvocation.MyCommand.Name)" }
}

#=============Source Server=============
Clear-Host
#if ($Host.Version.Major -ne 2) { throw "Use PowerShell Version 2"; break }

$Installed=Get-PSSnapin | Where-Object { $_.Name -eq "Microsoft.SharePoint.PowerShell" }
if ($Installed -eq $null) { Add-PSSnapin "Microsoft.SharePoint.PowerShell" }

$VerbosePreference="Continue"
$SourceWebURL="http://<sharepoint site>"
$SourceLibrary="<library name>"
$Parts=3
$ExportFolder="<fully qualified path>"

#=============Don't modify below=============

if (!(Test-Path -Path $ExportFolder -PathType Container) { throw "Bad Export Path" }
$SourceWeb=Get-SPWeb -Identity $SourceWebURL -Verbose:$false

$Alphabet=([char[]]([int][char]'A'..[int][char]'Z'))
$LibraryNameMod=0..($Parts-1) | ForEach-Object { $Alphabet[$_] }

if (!$ExportFolder.EndsWith("\")) { $ExportFolder+="\" }
if ($SourceWebURL.EndsWith("/")) { $SourceWebURL=$SourceWebURL.Substring(0,$SourceWebURL.Length-1) }

# Make sure template doesn't already exist
$EmptyTemplateName=$SourceLibrary+"Template"
Write-Verbose "Looking for pre-existing template $EmptyTemplateName"
$CustomListTemplate=$Sourceweb.Site.GetCustomListTemplates($SourceWeb) | Where-Object { $_.Name -eq $EmptyTemplateName }
if ($CustomListTemplate -ne $null) {
  Write-Verbose "Deleting Template: $EmptyTemplateName"
  $ListTemplates=$SourceWeb.Site.RootWeb.GetFolder("_catalogs/lt")
  $Template=$ListTemplates.Files | Where-Object { $_.Name -eq $CustomListTemplate.InternalName }
  $Template.Delete()
}
$SourceWeb.RecycleBin.DeleteAll()

# Save library as empty Template
Write-Verbose "Saving $SourceLibrary as empty template: $EmptyTemplateName"
$SourceWeb.Lists[$SourceLibrary].SaveAsTemplate($EmptyTemplateName, $EmptyTemplateName, $EmptyTemplateName, $false)

# Make sure empty libraries do not already exist
$LibraryNameMod | ForEach-Object { $LibraryName=$SourceLibrary+$_; $Library=$SourceWeb.Lists | Where-Object { $_.Title -eq $LibraryName }; if ($Library -ne $null) { Write-Verbose "Deleting pre-existing library: $LibraryName"; $Library.Delete() } }

# Create new temporary libraries from empty Template
$CustomListTemplate=$Sourceweb.Site.GetCustomListTemplates($Sourceweb) | Where-Object { $_.Name -eq $EmptyTemplateName }
$LibraryNameMod | ForEach-Object { $TemplateName=$SourceLibrary+$_; Write-Verbose "Creating empty library: $TemplateName"; $SourceWeb.Lists.Add($TemplateName,$TemplateName,$CustomListTemplate) > $null }

# Get all the folders in the source library so we can split it into $Parts parts
$SourceFolder=$SourceWeb.GetFolder($SourceLibrary)
$SubFolders=$SourceFolder.SubFolders #| Sort-Object -Property Name
$Number=[System.Convert]::ToInt32([System.Math]::Floor($SubFolders.Count/$Parts))
$Adjust=$SubFolders.Count-($SubFolders.Count-($Number*$Parts))
for ($i=0; $i -lt $Parts; $i++) {
  Write-Verbose "Processing $($SourceLibrary+$LibraryNameMod[$i])"
  $TargetFolder=$SourceWeb.GetFolder($SourceLibrary+$LibraryNameMod[$i])
  Write-Verbose "Processing $Number/$($Subfolders.Count) folders"
  for ($j=0; $j -lt $Number; $j++)  {
    $Index=($i*$Number)+$j
    Write-Verbose "Processing folder #$Index"
    Write-Verbose "Creating folder: $($SubFolders[$Index].Name)"
    $Folder=$TargetFolder.SubFolders.Add($SubFolders[$Index].Name)
    $Folder.Update()
    Copy-SPFiles -SourceFolder $SubFolders[$Index] -TargetFolder $Folder
  }
}
for ($i=$Adjust; $i -lt $SubFolders.Count; $i++) {
  Write-Verbose "Processing folder #$i"
  Write-Verbose "Creating folder: $($SubFolders[$i].Name)"
  $Folder=$TargetFolder.SubFolders.Add($SubFolders[$i].Name)
  $Folder.Update()
  Copy-SPFiles -SourceFolder $SubFolders[$i] -TargetFolder $Folder
}

# Make sure content templates don't already exist
for ($i=0; $i -lt $LibraryNameMod.Count; $i++) {
  $ContentTemplateName=$SourceLibrary+$LibraryNameMod[$i]
  Write-Verbose "Looking for pre-existing template $ContentTemplateName"
  $CustomListTemplate=$Sourceweb.Site.GetCustomListTemplates($SourceWeb) | Where-Object { $_.Name -eq $ContentTemplateName }
  if ($CustomListTemplate -ne $null) {
    Write-Verbose "Deleting Template: $ContentTemplateName"
    $ListTemplates=$SourceWeb.Site.RootWeb.GetFolder("_catalogs/lt")
    $Template=$ListTemplates.Files | Where-Object { $_.Name -eq $CustomListTemplate.InternalName }
    $Template.Delete()
  }
}
$SourceWeb.RecycleBin.DeleteAll()

# Save libraries as templates with content
$LibraryNameMod | ForEach-Object { $TemplateName=$SourceLibrary+$_; Write-Verbose "Creating Content Template: $TemplateName"; $SourceWeb.Lists[$TemplateName].SaveAsTemplate($TemplateName, $TemplateName, $TemplateName, $true) }
# Delete extra libraries
$LibraryNameMod | ForEach-Object { $LibraryName=$SourceLibrary+$_; Write-Verbose "Deleting Library: $LibraryName"; $SourceWeb.Lists[$LibraryName].Delete() }
$SourceWeb.RecycleBin.DeleteAll()

# Export content templates
# PowerShell 2 version
$WebRequest=New-Object System.Net.WebClient
$WebRequest.UseDefaultCredentials=$true
for ($i=0; $i -lt $LibraryNameMod.Count; $i++) {
  $ContentTemplateName=$SourceLibrary+$LibraryNameMod[$i]
  $CustomListTemplate=$Sourceweb.Site.GetCustomListTemplates($SourceWeb) | Where-Object { $_.Name -eq $ContentTemplateName }
  Write-Verbose "Exporting Content Template: $ContentTemplateName"
  $Data=$WebRequest.DownloadData("$SourceWebURL/_catalogs/lt/$($CustomListTemplate.InternalName)")
  [System.IO.File]::WriteAllBytes($ExportFolder+$CustomListTemplate.InternalName,$Data)
}
# PowerShell 5 version
#Invoke-WebRequest -Uri "http://share01-colo/sites/EC/_catalogs/lt/$($SourceLibrary)Template.stp" -OutFile "C:\Users\Administrator\Desktop\$($SourceLibrary)Template.stp" -UseDefaultCredentials -ContentType 
#$LibraryNameMod | ForEach-Object { $Junk=$SourceLibrary+$_; Write-Verbose "Exporting Library Content Template: $Junk"; Invoke-WebRequest -Uri "http://share01-colo/sites/EC/_catalogs/lt/$($SourceLibrary)Template.stp" -OutFile "C:\Users\Administrator\Desktop\$($SourceLibrary)Template.stp" -UseDefaultCredentials

# Delete Content Templates
for ($i=0; $i -lt $LibraryNameMod.Count; $i++) {
  $ContentTemplateName=$SourceLibrary+$LibraryNameMod[$i]
  $CustomListTemplate=$Sourceweb.Site.GetCustomListTemplates($SourceWeb) | Where-Object { $_.Name -eq $ContentTemplateName }
  Write-Verbose "Deleting Template: $ContentTemplateName"
  $ListTemplates=$SourceWeb.Site.RootWeb.GetFolder("_catalogs/lt")
  $Template=$ListTemplates.Files | Where-Object { $_.Name -eq $CustomListTemplate.InternalName }
  $Template.Delete()
}
$SourceWeb.RecycleBin.DeleteAll()

#=============Target Server=============
Clear-Host

$Installed=Get-PSSnapin | Where-Object { $_.Name -eq "Microsoft.SharePoint.PowerShell" }
if ($Installed -eq $null) { Add-PSSnapin "Microsoft.SharePoint.PowerShell" }

$VerbosePreference="Continue"
$TargetWebURL="http://<sharepoint site>"
$TargetLibrary="<library name>"
$ImportFolder="<fully qualified path>"

#=============Don't modify below=============

$TargetWeb=Get-SPWeb -Identity $TargetWebURL -Verbose:$false

if (!$ImportFolder.EndsWith("\")) { $ImportFolder+="\" }
if ($TargetWebURL.EndsWith("/")) { $TargetWebURL=$TargetWebURL.Substring(0,$TargetWebURL.Length-1) }

# Get templates to import, make sure they don't exist
$Files=Get-ChildItem -Path $ImportFolder
$Files=@($Files)
$LibraryNameMod=@()
$LibraryNameMod=$Files | ForEach-Object { $_.BaseName.Replace($TargetLibrary,"") }

$ListTemplateFiles=$TargetWeb.Site.RootWeb.GetFolder("_catalogs/lt")
$TargetSite=$TargetWeb.Site.OpenWeb()
$ListTemplates=$TargetSite.GetFolder("List Template Gallery")
$Templates=$ListTemplates.Files
for ($i=0; $i -lt $Files.Count; $i++) {
  Write-Verbose "Looking for pre-existing template $($Files[$i].BaseName)"
  $CustomListTemplate=$TargetWeb.Site.GetCustomListTemplates($TargetWeb) | Where-Object { ($_.InternalName -eq $Files[$i].Name) -or ($_.Name -eq $Files[$i].BaseName) }
  $CustomListTemplate | ForEach-Object { Write-Verbose "Deleting Template: $($_.Name)"; $TemplateInternalName=$_.InternalName; $Template=$ListTemplateFiles.Files | Where-Object { $_.Name -eq $TemplateInternalName }; $Template.Delete() }
  $TargetWeb.RecycleBin.DeleteAll()
  Write-Verbose "Adding Template: $($Files[$i].BaseName)"
  $Template=Get-Item -Path $Files[$i].FullName
  $Templates.Add("_catalogs/lt/$($Files[$i].Name)",$Template.OpenRead(),$true) > $null
}

# Make sure target library doesn't exist
$Library=$TargetWeb.Lists[$TargetLibrary]
if ($Library -ne $null) {
  Write-Verbose "Deleting Library: $TargetLibrary"
  $Library.Delete()
}

# Make sure extra libraries don't already exist
for ($i=0; $i -lt $LibraryNameMod.Count; $i++) {
  $LibraryName=$TargetLibrary+$LibraryNameMod[$i]
  Write-Verbose "Looking for $LibraryName"
  $Library=$TargetWeb.Lists[$LibraryName]
  if ($Library -ne $null) {
    Write-Verbose "Deleting Library: $LibraryName"
    $Library.Delete()
  }
}
$TargetWeb.RecycleBin.DeleteAll()

# Create base library from empty Template
$Template=$TargetWeb.Site.GetCustomListTemplates($TargetWeb) | Where-Object { $_.Name -eq ($TargetLibrary+"Template") }
Write-Verbose "Creating Library: $TargetLibrary"
$TargetWeb.Lists.Add($TargetLibrary,$TargetLibrary,$Template) > $null

# Create libraries from content Templates
$Templates=@()
# Get content templates...
$LibraryNameMod | ForEach-Object { $TemplateName=$_; $Templates+=$TargetWeb.Site.GetCustomListTemplates($TargetWeb) | Where-Object { $_.Name -eq ($TargetLibrary+$TemplateName) } }
# Create new library from template
$Templates | ForEach-Object { if (!$_.Name.EndsWith("Template")) { $TemplateName=$_.Name; Write-Verbose "Creating Library: $Junk"; $TargetWeb.Lists.Add($TemplateName,$TemplateName,$_) > $null } }

# Delete content templates
for ($i=0; $i -lt $Files.Count; $i++) {
  if (!$Files[$i].BaseName.EndsWith("Template")) {
    $ListTemplates=$TargetWeb.Site.RootWeb.GetFolder("_catalogs/lt")
    $Template=$ListTemplates.Files | Where-Object { $_.Name -eq $Files[$i].Name }
    Write-Verbose "Deleting Template: $($Template.Name)"
    $Template.Delete()
  }
}
$TargetWeb.RecycleBin.DeleteAll()

# Move documents from extra libraries back to original
$TargetFolder=$TargetWeb.GetFolder($TargetLibrary)
for ($i=0; $i -lt $LibraryNameMod.Count; $i++) {
  if ($LibraryNameMod -ne "Template") {
    $SourceFolder=$TargetWeb.GetFolder($TargetLibrary+$LibraryNameMod[$i])
    Write-Verbose "Copying $($SourceFolder.Name) to $($TargetFolder.Name)"
    Copy-SPFiles -SourceFolder $SourceFolder -TargetFolder $TargetFolder -Verbose:$false
  }
}

# Delete extra libraries
$LibraryNameMod | ForEach-Object { if ($_ -ne "Template") { $LibraryName=$TargetLibrary+$_; Write-Verbose "Deleting Library: $LibraryName"; $TargetWeb.Lists[$LibraryName].Delete() } }
$TargetWeb.RecycleBin.DeleteAll()

# The files are still in use for a bit, so we need to wait for the system to release them
$Template=$null
#Write-Verbose "Waiting 120 seconds for release of open file handles"
#Start-Sleep -Seconds 120
$Files | ForEach-Object { Write-Verbose "Deleting $($_.Name)"; Remove-Item -Path $_.FullName }