How do I migrate a document library (and just the documents) with PowerShell

Question

I need to migrate the documents of an existing library to a new library. The requirements of the migration is that I maintain versions and audit data (who/when created the document and last modified it).

The Export-SPWeb and Import-SPWeb cmdlets only seem to work on a list level or higher. This means that the old list gets created on import and includes custom views and what-not. But I want only the items to be migrated.

It is also my understanding that those cmdlets are ultimately using classes and methods in the Microsoft.SharePoint.Deployment namespace.

Does anybody know how to do this?

Update

I have been researching the Content Migration API and here is what I have found.

Exporting Just List Items

$ver = $host | select version
if($Ver.version.major -gt 1) {$Host.Runspace.ThreadOptions = "ReuseThread"}
if(!(Get-PSSnapin Microsoft.SharePoint.PowerShell -ea 0))
{
  Write-Progress -Activity "Loading Modules" -Status "Loading Microsoft.SharePoint.PowerShell"
  Add-PSSnapin Microsoft.SharePoint.PowerShell
}

[string]$sourceSiteUrl = "http://some.server.com"
[string]$sourceListName = "SomeList"

$fileLocation = $pwd


$site = New-Object Microsoft.SharePoint.SPSite($sourceSiteUrl)
$web = $site.OpenWeb()
$list = $web.Lists[$sourceListName]

$settings = New-Object Microsoft.SharePoint.Deployment.SPExportSettings
$settings.BaseFileName = "test.cmp"
$settings.CommandLineVerbose = $true
$settings.ExportMethod = [Microsoft.SharePoint.Deployment.SPExportMethodType]::ExportAll
$settings.FileCompression = $true
$settings.FileMaxSize = [Int32]::MaxValue
$settings.IncludeSecurity = [Microsoft.SharePoint.Deployment.SPIncludeSecurity]::All
$settings.IncludeVersions = [Microsoft.SharePoint.Deployment.SPIncludeVersions]::CurrentVersion
$settings.LogFilePath = "$pwd\log.txt"
$settings.OverwriteExistingDataFile = $true
$settings.SiteUrl = $sourceSiteUrl
$settings.FileLocation = $fileLocation

ForEach ($i In $list.Items) {
  $eo = New-Object Microsoft.SharePoint.Deployment.SPExportObject
  $eo.Id = $i.UniqueId
  $eo.Type = [Microsoft.SharePoint.Deployment.SPDeploymentObjectType]::ListItem

  $settings.ExportObjects.Add($eo)
}

$settings.Validate()

$export = New-Object Microsoft.SharePoint.Deployment.SPExport($settings)
$export.Run()

This script will export all items (including documents) from the specified list. The key is adding each item to the ExportObjects collection of the SPExportSettings object.

Importing Just List Items

$ver = $host | select version
if($Ver.version.major -gt 1) {$Host.Runspace.ThreadOptions = "ReuseThread"}
if(!(Get-PSSnapin Microsoft.SharePoint.PowerShell -ea 0))
{
  Write-Progress -Activity "Loading Modules" -Status "Loading Microsoft.SharePoint.PowerShell"
  Add-PSSnapin Microsoft.SharePoint.PowerShell
}

[string]$destSiteUrl = "http://some.server.com"    # or http://some.other.server.com
[string]$destListName = "SomeOtherList"

$fileLocation = $pwd

$site = New-Object Microsoft.SharePoint.SPSite($destSiteUrl)
$web = $site.OpenWeb()
$list = $web.Lists[$destListName]

$settings = New-Object Microsoft.SharePoint.Deployment.SPImportSettings
$settings.BaseFileName = "test.cmp"
$settings.CommandLineVerbose = $true
$settings.FileCompression = $true
$settings.LogFilePath = "$pwd\in-log.txt"
$settings.UserInfoDateTime = [Microsoft.SharePoint.Deployment.SPImportUserInfoDateTimeOption]::ImportAll
$settings.SiteUrl = $destSiteUrl
$settings.WebUrl = $destSiteUrl
$settings.FileLocation = $fileLocation
$settings.RetainObjectIdentity = $false

$settings.Validate()

$import = New-Object Microsoft.SharePoint.Deployment.SPImport($settings)

$oe = Register-ObjectEvent -InputObject $import -EventName Started -SourceIdentifier ImportStarted -Action {
  $ro = $eventArgs.RootObjects
  ForEach ($io in $ro) {
    if ($io.Type -eq [Microsoft.SharePoint.Deployment.PDeploymentObjectType]::ListItem) {
      $io.TargetParentUrl = $list.RootFolder.ServerRelativeUrl
    }
  }
}

$import.Run()

Get-EventSubscriber | Unregister-Event

This SHOULD perform the import on just the items but is not working correctly.

The problem seems to be around the Register-ObjectEvent code. When an object gets imported into SharePoint, the object needs to have a parent url. Because I want to import into a list other than the source list, I need to re-parent the list items. According to the document, I should be using the Started event of the SPImport object to find the "orphaned" objects and point them to the desired list. For some reason, the Started event is not being raised or if it is, PS is not executing my handler code.

Answer 1

Because I could not get an answer to the event handling in PowerShell, the posted Import PowerShell script does not work. As a solution, I merged the two scripts and ported them to C#. Here is the code.

using System;
using Microsoft.SharePoint;

namespace MigrateArticles {

   public class Program {

      #region Shared Constants

      protected static readonly string LogFile = System.IO.Path.Combine(Properties.Settings.Default.WorkFolder, "migration-log.txt");

      #endregion

      #region Private Methods

      protected static void ExportContent() {
         try {
            // clean up file location and re-create it
            if (System.IO.Directory.Exists(Properties.Settings.Default.WorkFolder)) {
               System.IO.Directory.Delete(Properties.Settings.Default.WorkFolder, true);
            }

            System.IO.Directory.CreateDirectory(Properties.Settings.Default.WorkFolder);

            var site = new Microsoft.SharePoint.SPSite(Properties.Settings.Default.SourceSiteUrl);
            var web = site.OpenWeb();
            var list = web.Lists[Properties.Settings.Default.SourceListName];

            // define the export settings
            var settings = new Microsoft.SharePoint.Deployment.SPExportSettings();
            settings.BaseFileName = Properties.Settings.Default.BaseFileName;
            settings.CommandLineVerbose = true;
            settings.ExportMethod = Microsoft.SharePoint.Deployment.SPExportMethodType.ExportAll;
            settings.FileCompression = false;
            settings.FileMaxSize = Int32.MaxValue;
            settings.IncludeSecurity = Microsoft.SharePoint.Deployment.SPIncludeSecurity.All;
            settings.IncludeVersions = Microsoft.SharePoint.Deployment.SPIncludeVersions.All;
            settings.LogFilePath = LogFile;
            settings.OverwriteExistingDataFile = true;
            settings.SiteUrl = Properties.Settings.Default.SourceSiteUrl;
            settings.FileLocation = Properties.Settings.Default.WorkFolder;

            // by default the entire site will be exported...
            //   loop through and specifically add each of the list items so only those get exported
            foreach (Microsoft.SharePoint.SPListItem i in list.Items) {
               var eo = new Microsoft.SharePoint.Deployment.SPExportObject();
               eo.Id = i.UniqueId;
               eo.Type = Microsoft.SharePoint.Deployment.SPDeploymentObjectType.ListItem;

               settings.ExportObjects.Add(eo);
            }

            // validate the settings, set up the export, and kick off the job
            settings.Validate();

            var export = new Microsoft.SharePoint.Deployment.SPExport(settings);

            export.Run();
         }
         catch (System.IO.FileNotFoundException ex) {
            Console.WriteLine();
            Console.WriteLine("The source site '{0}' was not found. Aborting migration.", Properties.Settings.Default.SourceSiteUrl);
            System.Environment.Exit(1);
         }
         catch (System.ArgumentNullException ex) {
            Console.WriteLine();
            Console.WriteLine("The source list '{0}' was not found. Aborting migration.", Properties.Settings.Default.SourceListName);
            System.Environment.Exit(1);
         }
         catch (System.Exception ex) {
            Console.WriteLine();
            Console.WriteLine("Unexpected exception during export. Aborting migration. Exception -- ", ex.Message);
            System.Environment.Exit(1);
         }
      }

      protected static void MapContent() {
         var manifestFileName = System.IO.Path.Combine(Properties.Settings.Default.WorkFolder, "Manifest.xml");

         // make sure manifest file exists
         if (!System.IO.File.Exists(manifestFileName)) {
            Console.WriteLine();
            Console.WriteLine("The 'manifest.xml' file does not exist. Aborting migration.");
            System.Environment.Exit(1);
         }

         // back it up!
         var manifestBackupFileName = System.IO.Path.Combine(Properties.Settings.Default.WorkFolder, "Manifest.ORIG.xml");
         System.IO.File.Copy(manifestFileName, manifestBackupFileName, true);

         // load it up
         var doc = new System.Xml.XmlDocument();
         doc.Load(manifestFileName);

         var root = doc.DocumentElement;

         var mgr = new System.Xml.XmlNamespaceManager(doc.NameTable);
         mgr.AddNamespace(@"x", @"urn:deployment-manifest-schema");

         // remove legacy content type
         var node = doc.SelectSingleNode(@"//x:SPObject[@Id='a2aa69e8-a1d8-0117-9744-146aeb47a14b']", mgr);
         if (node != null) {
            root.RemoveChild(node);
         }

         // remove folder
         node = doc.SelectSingleNode(@"//x:SPObject[@Id='cf3362c4-1e1b-43a5-8715-0e1a5a7b167']", mgr);
         if (node != null) {
            root.RemoveChild(node);
         }

         // remove site
         node = doc.SelectSingleNode(@"//x:SPObject[@Id='6430e1f7-e66f-4280-8b7c-91e15b57a29d']", mgr);
         if (node != null) {
            root.RemoveChild(node);
         }

         // write it back
         doc.Save(manifestFileName);
      }

      protected static void ImportContent() {
         try {
            var site = new Microsoft.SharePoint.SPSite(Properties.Settings.Default.DestinationSiteUrl);
            var web = site.OpenWeb();
            var list = web.Lists[Properties.Settings.Default.DestinationListName];

            var ct = list.ContentTypes[Properties.Settings.Default.ArticleContentTypeName];

            // define the import settings
            var settings = new Microsoft.SharePoint.Deployment.SPImportSettings();
            settings.BaseFileName = Properties.Settings.Default.BaseFileName;
            settings.CommandLineVerbose = true;
            settings.FileCompression = false;
            settings.LogFilePath = LogFile;
            settings.UserInfoDateTime = Microsoft.SharePoint.Deployment.SPImportUserInfoDateTimeOption.ImportAll;
            settings.SiteUrl = Properties.Settings.Default.DestinationSiteUrl;
            settings.WebUrl = Properties.Settings.Default.DestinationSiteUrl;
            settings.RetainObjectIdentity = false;
            settings.FileLocation = Properties.Settings.Default.WorkFolder;

            // validate settings, set up the import, define event handlers, and kick off the job
            settings.Validate();
            var importer = new Microsoft.SharePoint.Deployment.SPImport(settings);

            importer.Started += (sender, e) => {
               // we need to re-aprent each of the orphaned list items
               var ro = e.RootObjects;
               foreach (Microsoft.SharePoint.Deployment.SPImportObject io in ro) {
                  if (io.Type == Microsoft.SharePoint.Deployment.SPDeploymentObjectType.ListItem) {
                     io.TargetParentUrl = list.RootFolder.ServerRelativeUrl;
                  }
               }
            };

            importer.ObjectImported += (sender, e) => {
               // we need to change the Content Type of the list item after it has been added
               if (e.Type == Microsoft.SharePoint.Deployment.SPDeploymentObjectType.ListItem) {
                  var item = list.GetItemByUniqueId(e.TargetId);
                  if (item != null) {
                     item["ContentTypeId"] = ct.Id;
                     item.SystemUpdate();
                  }
               }
            };

            importer.Run();
         }
         catch (System.IO.FileNotFoundException ex) {
            Console.WriteLine();
            Console.WriteLine("The destination site '{0}' was not found. Aborting migration.", Properties.Settings.Default.DestinationSiteUrl);
            System.Environment.Exit(1);
         }
         catch (System.ArgumentNullException ex) {
            Console.WriteLine();
            Console.WriteLine("The detination list '{0}' was not found. Aborting migration.", Properties.Settings.Default.DestinationListName);
            System.Environment.Exit(1);
         }
         catch (System.Exception ex) {
            Console.WriteLine();
            Console.WriteLine("Unexpected exception during import. Aborting migration. Exception -- ", ex.Message);
            System.Environment.Exit(1);
         }
      }

      #endregion

      #region Public Methods

      public static void Main(string[] args) {
         if (Properties.Settings.Default.DoExport) {
            ExportContent();
         }

         if (Properties.Settings.Default.DoImport) {
            MapContent();
            ImportContent();
         }
      }

      #endregion

   }

}

Answer 2

The PowerShell Register-ObjectEvent adds events to a queue and processes them asynchronously, i.e. after the import has run, so it changes the value of the parent too late to be of any use. Also, there is a typo in "PDeploymentObjectType".

You can add a synchronous event handler with the following:

$import.add_Started({
    param($Source, $EventArgs)
    Write-Host "SYNC EVENT: Import Started"
    Write-Host "SYNC Value of list: '$($list.RootFolder.ServerRelativeUrl)'"
    $ro = $EventArgs.RootObjects
    ForEach ($io in $ro) {
        if ($io.Type `
            -eq [Microsoft.SharePoint.Deployment.SPDeploymentObjectType]::ListItem) {
            $io.TargetParentUrl = $list.RootFolder.ServerRelativeUrl
        }
    }
})

Also add output from the Register-ObjectEvent to see how it fires later. You will also see that the $list variable has the value from the shell (usually $null), not the script.

$oe = Register-ObjectEvent -InputObject $import -EventName Started `
    -SourceIdentifier ImportStarted -Action {
    Write-Host "ASYNC EVENT: Import Started, happens after event has completed"
    Write-Host "ASYNC Value of list: '$($list.RootFolder.ServerRelativeUrl)'"
}

Answer 3

Jason,

Can you please try the suggestion given in following question:

How can you move a document with its version history to a new library?

It says

Just moved (drag & drop) a file in Explorer View from one document library to another, and it did retain previous versions!

Obviously both libraries need to have versioning enabled.

Try it yourself.

EDIT: note that it ONLY retains the versions if you MOVE the files. It does not do so when you perform a COPY!