3

I have a site collection with a dozen or so document libraries in it (total size about 300GB's). The goal is to move each of these document libraries into it's own site collection (and content database).

I've seen posts like Exporting sites to a new content database (talks about moving whole sites) and Restructuring a Very Large Document Library (restructuring a large document library). I found an article online that specifically adresses this issue, http://pravahaminfo.blogspot.com/2011/05/how-to-move-document-library-from-one.html, but the method of exporting the list as a template with content and then importing it on the new site only works up to 10 MB's (500 Mb's max if you change it) and my smallest document library is about 5GB's.

Right now I am considering exporting the list as a template WITHOUT content, importing it, and then using code to move the files. I looked into using Explorer view to move the files but the document libraries I have with over 20,000 folders in the root won't open in Explorer view...

Suggestions on how I should move forward? I'm currently looking for a method that will work in 2007 BUT we are migrating to 2010 in the near future so if the time/headache savings are big enough I will wait.

1
  • Any update on this? Is the best method to export somehow in v3 and import into v4, or is it better to upgrade the db to v4 and then use Export-SPWeb? I'm doing the same thing but ultimately going from v3 to v5, but I have a v4 environment at my disposal as well. Does that change the equation at all?
    – thanby
    May 30 '13 at 2:43
3

I am going to provide an answer that will work in 2010. I will need to depend on others who are more versed in MOSS for that; however, I believe there is a similar stsadm command for the Powershell I will provide below.

The Export-SPWeb command is able to be defined for a site, list or library. I have used this extensively to move a troubling list from production to dev/test so that I may work on a specific criteria.

Here is the TechNet Article

An example:

Add-PSSnapin Microsoft.SharePoint.PowerShell
$URL = "http://iwanttotest.com/MyManagedPath/SiteName"
$DocLibrary = "http://iwanttotest.com/apps/site/DocLibraryName"
Export-SPWeb -Identity $URL -Path C:\SiteBackup.cmp -ItemURL $DocLibrary -IncludeUserSecurity -IncludeVersions

Please note this must be executed on a server within the SP Farm so you may add the Powershell snapin.

4
  • Exporting lists and libraries is was added in SP2010 and is not possible with the export stsadm operation in 2007. Upgrading first to 2010 and then exporting is a good solution provided Patricker can wait. SharePoint upgrades are also tricky in their own right.
    – shufler
    Nov 14 '11 at 21:54
  • Thanks for the assist shufler. I found the upgrade to 2010 to be fairly good, but I scripted every piece with Powershell and ran the upgrade and revert likely 100 times before the live execution.
    – websch01ar
    Nov 15 '11 at 12:10
  • I did some more research, what about using Microsoft.SharePoint.Deployment.SPExport / SPImport? This appears to be able handle SPList objects.
    – Peter
    Nov 15 '11 at 15:25
  • My assumption is stsadm wraps SPExport. Obviously that doesn't mean SPExport can't handle SPLists, but I'm not seeing anything other than SPSites mentioned in the MSDN for either v3 or v4.
    – shufler
    Nov 16 '11 at 0:11
1

Introduction

I had to move a document library which was bigger than the 524,288,000 byte maximum from an old SharePoint 2010 to SharePoint 2019. So, I came up with this solution. It's not perfect, but it works. Please don't run it blind as it could really mess up your server.

Caveats

  • My source is a SharePoint 2010, my target is SharePoint 2019. This means that I ran the Source part of the script on PowerShell 2 and the Target part on PowerShell 5.
  • I ran this solution from an account with full administrative credentials in SharePoint.
  • My library had no root files, only folders.
  • I was unable to run $Site=Get-SPSite "http://<sharepoint site>"; $Site.RecalculateStorageMetrics(), so I was unable to calculate any sizes. Therefore, I had to guess as to how many parts to break it into to get a size smaller than the maximum.
  • The function Copy-SPFiles is a prerequisite for both the Source and the Target sections.

Overview

Source:

  1. Create an empty template of the library I want to export
  2. Create $Parts number of new libraries from the empty template
  3. Determine the integer number of folders in each part, and save the remainder as $Adjust
  4. Copy $Parts number of folders in sequence into each sub-library
  5. Copy $Adjust number of folders in sequence into the last library
  6. Save sub-libraries as Templates with content preserved
  7. Delete sub-libraries
  8. Export the Content Templates to a file
  9. Delete the Content Templates

Target:

  1. Determine the number of files to import
  2. Import each file into the List Template Gallery
  3. Create the libraries from the templates
  4. Delete the Content Templates (Keep the empty one)
  5. Copy the files from the sub-libraries into the target library
  6. Delete the sub-libraries
  7. Delete the files

Code

function Copy-SPFiles {
  [CmdletBinding(SupportsShouldProcess=$false)]
  Param ([Parameter(Mandatory=$true, ValueFromPipeline=$false)] [ValidateNotNullOrEmpty()] [Microsoft.SharePoint.SPFolder]$SourceFolder,
          [Parameter(Mandatory=$true, ValueFromPipeline=$false)] [ValidateNotNullOrEmpty()] [Microsoft.SharePoint.SPFolder]$TargetFolder,
          [Parameter(Mandatory=$false, ValueFromPipeline=$false)] [int]$MaxRootFoldersToCopy=[System.Int32]::MaxValue,
          [Parameter(Mandatory=$false, ValueFromPipeline=$false)] [switch]$BeyondRoot)
          #[Parameter(Mandatory=$false, ValueFromPipeline=$false, DontShow=$true)] [switch]$BeyondRoot)

  Begin { Write-Verbose "Entering $($PSCmdlet.MyInvocation.MyCommand.Name)" }

  Process {
    Write-Verbose "Processing $($SourceFolder.Files.Count.ToString()) files"
    for ($i=0; $i -lt $SourceFolder.Files.Count; $i++) {
      Write-Verbose "Processing File: $($SourceFolder.Files[$i].Name)"
      $NewFile = $TargetFolder.Files.Add($SourceFolder.Files[$i].Name, $SourceFolder.Files[$i].OpenBinary(),$True)

      # Copy Meta-Data from Source
      for ($j=0; $j -lt $SourceFolder.Files[$i].Item.Fields.Count; $j++) {
        if (!$SourceFolder.Files[$i].Item.Fields[$j].ReadOnlyField) {
          if ($NewFile.Item.Fields.ContainsField($SourceFolder.Files[$i].Item.Fields[$j].InternalName)) {
          $NewFile.Item[$SourceFolder.Files[$i].Item.Fields[$j].InternalName]=$SourceFolder.Files[$i].Item[$SourceFolder.Files[$i].Item.Fields[$j].InternalName]
          }
        }
      }
      $NewFile.Item.UpdateOverwriteVersion()
    }
 
    #Process SubFolders
    Write-Verbose "Processing $($SourceFolder.SubFolders.Count.ToString()) folders"
    if ($BeyondRoot.IsPresent) { $Limit=$SourceFolder.SubFolders.Count }
    else {
      if ($MaxRootFoldersToCopy -lt $SourceFolder.SubFolders.Count) { $Limit=$MaxRootFoldersToCopy } else { $Limit=$SourceFolder.SubFolders.Count }
      if ($Limit -gt $SourceFolder.SubFolders.Count) { $Limit=$SourceFolder.SubFolders.Count }
    }
    if (!$BeyondRoot.IsPresent) { Write-Verbose "But only up to $Limit" }
    for ($i=0; $i -lt $Limit; $i++) {
      Write-Verbose "Processing Folder: $($SourceFolder.SubFolders[$i].Name)"
      if ($SourceFolder.SubFolders[$i].Name -ne "Forms") {
        # Check if Sub-Folder exists in the Target Library
        $NewTargetFolder = $TargetFolder.ParentWeb.GetFolder($SourceFolder.SubFolders[$i].Name)
        if (!$NewTargetFolder.Exists) {
          $NewTargetFolder = $TargetFolder.SubFolders.Add($SourceFolder.SubFolders[$i].Name)
        }
        # Call the function recursively
        Copy-SPFiles -SourceFolder $SourceFolder.SubFolders[$i] -TargetFolder $NewTargetFolder -MaxRootFoldersToCopy $MaxRootFoldersToCopy -BeyondRoot
      }
    }
  }

  End { Write-Verbose "Exiting $($PSCmdlet.MyInvocation.MyCommand.Name)" }
}

#=============Source Server=============
Clear-Host
#if ($Host.Version.Major -ne 2) { throw "Use PowerShell Version 2"; break }

$Installed=Get-PSSnapin | Where-Object { $_.Name -eq "Microsoft.SharePoint.PowerShell" }
if ($Installed -eq $null) { Add-PSSnapin "Microsoft.SharePoint.PowerShell" }

$VerbosePreference="Continue"
$SourceWebURL="http://<sharepoint site>"
$SourceLibrary="<library name>"
$Parts=3
$ExportFolder="<fully qualified path>"

#=============Don't modify below=============

if (!(Test-Path -Path $ExportFolder -PathType Container) { throw "Bad Export Path" }
$SourceWeb=Get-SPWeb -Identity $SourceWebURL -Verbose:$false

$Alphabet=([char[]]([int][char]'A'..[int][char]'Z'))
$LibraryNameMod=0..($Parts-1) | ForEach-Object { $Alphabet[$_] }

if (!$ExportFolder.EndsWith("\")) { $ExportFolder+="\" }
if ($SourceWebURL.EndsWith("/")) { $SourceWebURL=$SourceWebURL.Substring(0,$SourceWebURL.Length-1) }

# Make sure template doesn't already exist
$EmptyTemplateName=$SourceLibrary+"Template"
Write-Verbose "Looking for pre-existing template $EmptyTemplateName"
$CustomListTemplate=$Sourceweb.Site.GetCustomListTemplates($SourceWeb) | Where-Object { $_.Name -eq $EmptyTemplateName }
if ($CustomListTemplate -ne $null) {
  Write-Verbose "Deleting Template: $EmptyTemplateName"
  $ListTemplates=$SourceWeb.Site.RootWeb.GetFolder("_catalogs/lt")
  $Template=$ListTemplates.Files | Where-Object { $_.Name -eq $CustomListTemplate.InternalName }
  $Template.Delete()
}
$SourceWeb.RecycleBin.DeleteAll()

# Save library as empty Template
Write-Verbose "Saving $SourceLibrary as empty template: $EmptyTemplateName"
$SourceWeb.Lists[$SourceLibrary].SaveAsTemplate($EmptyTemplateName, $EmptyTemplateName, $EmptyTemplateName, $false)

# Make sure empty libraries do not already exist
$LibraryNameMod | ForEach-Object { $LibraryName=$SourceLibrary+$_; $Library=$SourceWeb.Lists | Where-Object { $_.Title -eq $LibraryName }; if ($Library -ne $null) { Write-Verbose "Deleting pre-existing library: $LibraryName"; $Library.Delete() } }

# Create new temporary libraries from empty Template
$CustomListTemplate=$Sourceweb.Site.GetCustomListTemplates($Sourceweb) | Where-Object { $_.Name -eq $EmptyTemplateName }
$LibraryNameMod | ForEach-Object { $TemplateName=$SourceLibrary+$_; Write-Verbose "Creating empty library: $TemplateName"; $SourceWeb.Lists.Add($TemplateName,$TemplateName,$CustomListTemplate) > $null }

# Get all the folders in the source library so we can split it into $Parts parts
$SourceFolder=$SourceWeb.GetFolder($SourceLibrary)
$SubFolders=$SourceFolder.SubFolders #| Sort-Object -Property Name
$Number=[System.Convert]::ToInt32([System.Math]::Floor($SubFolders.Count/$Parts))
$Adjust=$SubFolders.Count-($SubFolders.Count-($Number*$Parts))
for ($i=0; $i -lt $Parts; $i++) {
  Write-Verbose "Processing $($SourceLibrary+$LibraryNameMod[$i])"
  $TargetFolder=$SourceWeb.GetFolder($SourceLibrary+$LibraryNameMod[$i])
  Write-Verbose "Processing $Number/$($Subfolders.Count) folders"
  for ($j=0; $j -lt $Number; $j++)  {
    $Index=($i*$Number)+$j
    Write-Verbose "Processing folder #$Index"
    Write-Verbose "Creating folder: $($SubFolders[$Index].Name)"
    $Folder=$TargetFolder.SubFolders.Add($SubFolders[$Index].Name)
    $Folder.Update()
    Copy-SPFiles -SourceFolder $SubFolders[$Index] -TargetFolder $Folder
  }
}
for ($i=$Adjust; $i -lt $SubFolders.Count; $i++) {
  Write-Verbose "Processing folder #$i"
  Write-Verbose "Creating folder: $($SubFolders[$i].Name)"
  $Folder=$TargetFolder.SubFolders.Add($SubFolders[$i].Name)
  $Folder.Update()
  Copy-SPFiles -SourceFolder $SubFolders[$i] -TargetFolder $Folder
}

# Make sure content templates don't already exist
for ($i=0; $i -lt $LibraryNameMod.Count; $i++) {
  $ContentTemplateName=$SourceLibrary+$LibraryNameMod[$i]
  Write-Verbose "Looking for pre-existing template $ContentTemplateName"
  $CustomListTemplate=$Sourceweb.Site.GetCustomListTemplates($SourceWeb) | Where-Object { $_.Name -eq $ContentTemplateName }
  if ($CustomListTemplate -ne $null) {
    Write-Verbose "Deleting Template: $ContentTemplateName"
    $ListTemplates=$SourceWeb.Site.RootWeb.GetFolder("_catalogs/lt")
    $Template=$ListTemplates.Files | Where-Object { $_.Name -eq $CustomListTemplate.InternalName }
    $Template.Delete()
  }
}
$SourceWeb.RecycleBin.DeleteAll()

# Save libraries as templates with content
$LibraryNameMod | ForEach-Object { $TemplateName=$SourceLibrary+$_; Write-Verbose "Creating Content Template: $TemplateName"; $SourceWeb.Lists[$TemplateName].SaveAsTemplate($TemplateName, $TemplateName, $TemplateName, $true) }
# Delete extra libraries
$LibraryNameMod | ForEach-Object { $LibraryName=$SourceLibrary+$_; Write-Verbose "Deleting Library: $LibraryName"; $SourceWeb.Lists[$LibraryName].Delete() }
$SourceWeb.RecycleBin.DeleteAll()

# Export content templates
# PowerShell 2 version
$WebRequest=New-Object System.Net.WebClient
$WebRequest.UseDefaultCredentials=$true
for ($i=0; $i -lt $LibraryNameMod.Count; $i++) {
  $ContentTemplateName=$SourceLibrary+$LibraryNameMod[$i]
  $CustomListTemplate=$Sourceweb.Site.GetCustomListTemplates($SourceWeb) | Where-Object { $_.Name -eq $ContentTemplateName }
  Write-Verbose "Exporting Content Template: $ContentTemplateName"
  $Data=$WebRequest.DownloadData("$SourceWebURL/_catalogs/lt/$($CustomListTemplate.InternalName)")
  [System.IO.File]::WriteAllBytes($ExportFolder+$CustomListTemplate.InternalName,$Data)
}
# PowerShell 5 version
#Invoke-WebRequest -Uri "http://share01-colo/sites/EC/_catalogs/lt/$($SourceLibrary)Template.stp" -OutFile "C:\Users\Administrator\Desktop\$($SourceLibrary)Template.stp" -UseDefaultCredentials -ContentType 
#$LibraryNameMod | ForEach-Object { $Junk=$SourceLibrary+$_; Write-Verbose "Exporting Library Content Template: $Junk"; Invoke-WebRequest -Uri "http://share01-colo/sites/EC/_catalogs/lt/$($SourceLibrary)Template.stp" -OutFile "C:\Users\Administrator\Desktop\$($SourceLibrary)Template.stp" -UseDefaultCredentials

# Delete Content Templates
for ($i=0; $i -lt $LibraryNameMod.Count; $i++) {
  $ContentTemplateName=$SourceLibrary+$LibraryNameMod[$i]
  $CustomListTemplate=$Sourceweb.Site.GetCustomListTemplates($SourceWeb) | Where-Object { $_.Name -eq $ContentTemplateName }
  Write-Verbose "Deleting Template: $ContentTemplateName"
  $ListTemplates=$SourceWeb.Site.RootWeb.GetFolder("_catalogs/lt")
  $Template=$ListTemplates.Files | Where-Object { $_.Name -eq $CustomListTemplate.InternalName }
  $Template.Delete()
}
$SourceWeb.RecycleBin.DeleteAll()

#=============Target Server=============
Clear-Host

$Installed=Get-PSSnapin | Where-Object { $_.Name -eq "Microsoft.SharePoint.PowerShell" }
if ($Installed -eq $null) { Add-PSSnapin "Microsoft.SharePoint.PowerShell" }

$VerbosePreference="Continue"
$TargetWebURL="http://<sharepoint site>"
$TargetLibrary="<library name>"
$ImportFolder="<fully qualified path>"

#=============Don't modify below=============

$TargetWeb=Get-SPWeb -Identity $TargetWebURL -Verbose:$false

if (!$ImportFolder.EndsWith("\")) { $ImportFolder+="\" }
if ($TargetWebURL.EndsWith("/")) { $TargetWebURL=$TargetWebURL.Substring(0,$TargetWebURL.Length-1) }

# Get templates to import, make sure they don't exist
$Files=Get-ChildItem -Path $ImportFolder
$Files=@($Files)
$LibraryNameMod=@()
$LibraryNameMod=$Files | ForEach-Object { $_.BaseName.Replace($TargetLibrary,"") }

$ListTemplateFiles=$TargetWeb.Site.RootWeb.GetFolder("_catalogs/lt")
$TargetSite=$TargetWeb.Site.OpenWeb()
$ListTemplates=$TargetSite.GetFolder("List Template Gallery")
$Templates=$ListTemplates.Files
for ($i=0; $i -lt $Files.Count; $i++) {
  Write-Verbose "Looking for pre-existing template $($Files[$i].BaseName)"
  $CustomListTemplate=$TargetWeb.Site.GetCustomListTemplates($TargetWeb) | Where-Object { ($_.InternalName -eq $Files[$i].Name) -or ($_.Name -eq $Files[$i].BaseName) }
  $CustomListTemplate | ForEach-Object { Write-Verbose "Deleting Template: $($_.Name)"; $TemplateInternalName=$_.InternalName; $Template=$ListTemplateFiles.Files | Where-Object { $_.Name -eq $TemplateInternalName }; $Template.Delete() }
  $TargetWeb.RecycleBin.DeleteAll()
  Write-Verbose "Adding Template: $($Files[$i].BaseName)"
  $Template=Get-Item -Path $Files[$i].FullName
  $Templates.Add("_catalogs/lt/$($Files[$i].Name)",$Template.OpenRead(),$true) > $null
}

# Make sure target library doesn't exist
$Library=$TargetWeb.Lists[$TargetLibrary]
if ($Library -ne $null) {
  Write-Verbose "Deleting Library: $TargetLibrary"
  $Library.Delete()
}

# Make sure extra libraries don't already exist
for ($i=0; $i -lt $LibraryNameMod.Count; $i++) {
  $LibraryName=$TargetLibrary+$LibraryNameMod[$i]
  Write-Verbose "Looking for $LibraryName"
  $Library=$TargetWeb.Lists[$LibraryName]
  if ($Library -ne $null) {
    Write-Verbose "Deleting Library: $LibraryName"
    $Library.Delete()
  }
}
$TargetWeb.RecycleBin.DeleteAll()

# Create base library from empty Template
$Template=$TargetWeb.Site.GetCustomListTemplates($TargetWeb) | Where-Object { $_.Name -eq ($TargetLibrary+"Template") }
Write-Verbose "Creating Library: $TargetLibrary"
$TargetWeb.Lists.Add($TargetLibrary,$TargetLibrary,$Template) > $null

# Create libraries from content Templates
$Templates=@()
# Get content templates...
$LibraryNameMod | ForEach-Object { $TemplateName=$_; $Templates+=$TargetWeb.Site.GetCustomListTemplates($TargetWeb) | Where-Object { $_.Name -eq ($TargetLibrary+$TemplateName) } }
# Create new library from template
$Templates | ForEach-Object { if (!$_.Name.EndsWith("Template")) { $TemplateName=$_.Name; Write-Verbose "Creating Library: $Junk"; $TargetWeb.Lists.Add($TemplateName,$TemplateName,$_) > $null } }

# Delete content templates
for ($i=0; $i -lt $Files.Count; $i++) {
  if (!$Files[$i].BaseName.EndsWith("Template")) {
    $ListTemplates=$TargetWeb.Site.RootWeb.GetFolder("_catalogs/lt")
    $Template=$ListTemplates.Files | Where-Object { $_.Name -eq $Files[$i].Name }
    Write-Verbose "Deleting Template: $($Template.Name)"
    $Template.Delete()
  }
}
$TargetWeb.RecycleBin.DeleteAll()

# Move documents from extra libraries back to original
$TargetFolder=$TargetWeb.GetFolder($TargetLibrary)
for ($i=0; $i -lt $LibraryNameMod.Count; $i++) {
  if ($LibraryNameMod -ne "Template") {
    $SourceFolder=$TargetWeb.GetFolder($TargetLibrary+$LibraryNameMod[$i])
    Write-Verbose "Copying $($SourceFolder.Name) to $($TargetFolder.Name)"
    Copy-SPFiles -SourceFolder $SourceFolder -TargetFolder $TargetFolder -Verbose:$false
  }
}

# Delete extra libraries
$LibraryNameMod | ForEach-Object { if ($_ -ne "Template") { $LibraryName=$TargetLibrary+$_; Write-Verbose "Deleting Library: $LibraryName"; $TargetWeb.Lists[$LibraryName].Delete() } }
$TargetWeb.RecycleBin.DeleteAll()

# The files are still in use for a bit, so we need to wait for the system to release them
$Template=$null
#Write-Verbose "Waiting 120 seconds for release of open file handles"
#Start-Sleep -Seconds 120
$Files | ForEach-Object { Write-Verbose "Deleting $($_.Name)"; Remove-Item -Path $_.FullName }

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.