Restructuring a Very Large Document Library

Question

Another question about the Very Large List I referenced in this question: http://sharepoint.stackexchange.comquestions/445/deleting-old-versions-in-a-document-library

This Document Library has over 5000 items and has outgrown its britches. The Document Library is an artifact repository for the SDLC process at one of my clients. Given this, each document belows to a specific project, which is specified in the item's metadata.

All of the documents currently live in the root of the Document Library. What we'd like to do is move the documents into folders by project to both improve performance as well as to reduce the likelihood of filename repetition causing destruction by overwrite.

This is a "no managed code" environment (don't ask), so I'm trying to use SharePoint's Lists and Copy Web Services to move the documents. The Copy Web Service copies the docs just fine, but we want to preserve the Created and Created By column values if at all possible. I've tried to set the Created and Created By columns ReadOnly=False to update them using the Lists Web Service's UpdateListItems operation, but can't make that work.

Any ideas?

Update 2009-11-12 - The Copy Web Service seems to preserve the Created and Created By column values just fine. Turns out that the last hurdle is unlinking the copy from its source. I can't figure out a way to empty out the _CopySource column. No combination of setting things like ReadOnly="False" on the column, etc. seems to work. This is one capability that I would expect to be part of the Copy Web Service, as it's exposed in the UI on the page when you look at the newly created item. If I delete the source item, then the _CopySource is still there, but with an invalid link back.

Answer 1

What about doing it manually using the explorer view?

You might want to update the filenames through the web services to prefix the files with the projectname (I hope you can perform something like a SystemUpdate() that preserves the user and datetime), so that you can easily select all files for a project (sort by filename) and move them in one go.

When all files are moved, rename back again using the webservices.

Answer 2

Marc,

Don't know if these stsadm extensions would be off limits to you or not, but you may find using them you can create some scripts to move the data for you based on the metadata? I would start here if you are able to use them. http://stsadm.blogspot.com/2007/08/stsadm-commands_09.html

Lori

Answer 3

Marc, is it an option to split the documentlibrary into a library per project as this would do wonders performancewise ?

Answer 4

After seeing my tweets about this, Peter Senescu at Metavis Technology suggested I look at their MetaVis Architect for SharePoint product. I've fired up the trial version, and it looks like it will get me where I want to go on this. I have no association with Metavis, and I've got to say that the product is really cool. It's going to let me move large groups of documents in one go based on their metadata values, which is exactly what I wanted to do. The one drawback is that we will lose the original Created and Created By column values, but we've decided that is an OK price to pay.

One nice thing about this tool is that there's no server-side install, which was a must-have in this case. It looks like it uses the Web Services (my recent favorite domain!) to get things done. I was trying to build my own Web Serivce-based solution, but I was never going to get anywhere near this level of sophistication.