SharePoint 2010 design workflow to run on entire library

Question

Summary - I have a library with MANY documents (over ten million) that I need to break up into directories. I have a List workflow designed using the directions from this page which works correctly, tho I only seem to be able to run it from records I select through the UI, and it only allows me to select one record.

What I really need is the ability to tell the workflow to iterate through the whole library without me having to check off the documents. I know that'll take quite a while, but it should still be faster than me checking off a couple hundred at a time and running the task.

I see a number of pages on how to build a for-next loop, cause apparently one doesn't exist in the workflow language, but before I go down that rabbit hole I want to make sure there isn't some other method - making it a site workflow, or a reusable, or some other option I don't even know about.

Answer 1

It's possible to achieve with a list workflow, though it certainly sounds like a treacherous rabbit hole to me. A site workflow might be better suited, but that would require developing your own actions in Visual Studio, since the default actions for site workflows in SP2010 do not include looping through list items (or even invoking web services!). Some third party workflow add-ons, like those offered by K2 and Nintex, add a fancier workflow interface and more custom actions that could easily achieve what you're looking for, but for a one-time update like this, they seem like overkill.

If you have access to log in to the web front end servers, a custom Powershell script will be the path of least resistance. Just start the SharePoint Management Shell (Start -> All Programs -> Microsoft SharePoint 2010 Products -> SharePoint 2010 Management Shell) and you can use Powershell to directly and programmatically access the SharePoint object model.

$web = get-spweb http://server/site
$list = $web.lists["Your Library Name"]
$list.Items | foreach-object { $file = $_.file; $file.MoveTo("destination url"); $file.Update();}
$web.dispose()

Instead of typing the commands directly into the console, you can save a complete script to a text file with the extension .ps1 and then execute it from the management shell by typing &"filename.ps1" (replacing filename.ps1 with the path to your file, which you can get by holding shift and right-clicking on the file, then selecting "copy as path").

Note that if you have exceeded the list view threshold, accessing $list.Items will likely fail due to size. You can temporarily disable throttling on the list by executing $list.EnableThrottling = $false. Alternatively, you can query the list using an SPQuery object to access a smaller subset of the items at a time, paging through the results until the entire list is processed.

I'm guessing you'll need a more complicated Powershell script than the barebones example above, but I recommend asking a new question for assistance (after searching for any relevant existing questions) if you do decide to go that route.

Answer 2

Using Thriggle's suggestion, and finally getting the syntax for the search down, I came up with a script that creates the folder if needed, and migrates the searched records to said folder. Shared below - suggestions for tweaks welcome.

Works in the test environment - keen to see how quickly it runs in Production against lists with millions of records.

Thanks to all - Share And Enjoy.

#######
# Update variables 

$SPListName = "TheListName"
$SPListDate = "2/22/2016"
$SPDirName = "2016_02"

$SPListPath = "http://Address.com/reps/historic/TheListName/"
######

$SPSite = Get-SPSite http://Address.com/reps/historic/

$SPWeb = $SPSite.openWeb();

$SPList = $SPWeb.Lists[$SPListName]

$testFolder = $SPWeb.GetFolder($SPListPath + $SPDirName)
 if ($testFolder.Exists)
 {
    #Do nothing
 }
 else
 {
    $folder = $SPList.AddItem("", Microsoft.SharePoint.SPFileSystemObjectType]::Folder, "$SPDirName")
    $folder.Update()
    Write-Host("Folder ${SPDirName} created");
 }

$SPItems = $SPList.Items | Where { $_["Processing_Date"] -eq $SPListDate }

$SPItems | foreach-object { $file = $_.file; $file.MoveTo($SPListPath +     $SPDirName + "/" + $file.Name); $file.Update();}

$SPCount = $SPItems.count.ToString()

Write-Host("${SPCount} records migrated");

$SPWeb.dispose()

Answer 3

I would recommend exploiting the information management policy settings. If you build a workflow that will shuttle the file to the appropriate library, as you seem to have done. Implement a IMP for expiration that runs on a created date +5 (or whatever) that calls that workflow.

I would recommend doing it in chunks. Make the date range (because you can only use dates) comprise a subset of the documents (10%?). Once that is complete, change the date and run it again. You can wait for it to run via timer job, or force run the timer job per this article.

Be advised that this will only run once per document.