Experience with storing 250.000 items in a Sharepoint 2010 Document Library without folders

Question

I want to store about 250.000 PDFs (average size of one PDF: about 1,4 MB) in a SharePoint 2010 Document Library. Does somebody have experience with this amout of items without folders in a single Document Library ?

My questions are especially: 1. Is storing this amout of items flat (without folders) officially supported by Microsoft ? (the only data i found is that 30 million items per document library is supported, but it doesn’t explicitly say if this amount is only possible/supported with folders [source: http://technet.microsoft.com/en-us/library/cc262787.aspx]) 2. One of our solution architects expressed concerns that some items might get corrupted when storing this amount of items flat, do you have experience with that ? 3. Are there any other reasons you would't recommend storing this amount of items flat ?

Peformance is not a primary conern. The items would be filtered with views

Best regards, Markus

Answer 1

If performance is not an issue then go ahead and go for it. Those limitations are for unnecessary trips to sql and fetch. But thats not an issue then go for it.

Answer 2

There is a misconception about the size of the list and the performance of the list.

In my own testing, I have found that list performance is fairly consistent from 101 items to 300,000 to 600,000 items. There is no discernible performance impact. To test this, I ran performance tests against a list with randomly generated items at intervals and recorded three runs at each interval for the page load speed (with an initial "prime" of the list view). I also took care to page three times forward to ensure that I wasn't benefiting from cached output.

While Microsoft has a recommendation of 2000 items per list view as a soft boundary for "acceptable performance", by default, when you load the list in the browser, you're looking at a view with 100 items at a time unless you change the number of items in the view.

What this means is that you will likely not see any performance impact, from a browser/user perspective, even with hundreds of thousands of items in your list if your list is primarily view/insert.

Admittedly, my tests were lacking. There was no testing of concurrency (multiple users accessing the list view), no testing of the effect of updates to the list (multiple users, some updating, some deleting, some inserting), no testing of column count on performance, and no testing of whole list update operations (i.e. adding a new field with a default value).

On a whole, if you just need to have a large list that is mostly being accessed with read and insert operations, you're probably fine with 250K, 500K, or even 1M.

Answer 3

You might want to have a look at external blob storage: http://msdn.microsoft.com/en-us/library/bb802976.aspx

In general it should be ok to use large lists/document libraries if you use the proper data access methods. Have a look at this Microsoft whitepaper which refers to a page load time of 3.5 s for a page working on a list with 100.000 items (retrieving 100 list items at a time): http://go.microsoft.com/fwlink/?LinkId=95450&clcid=0x409

You also might want to consider the content DB size. A large content DB is unhandy for backup and restore operations depending on what hardware is being used. I think 100 GB is recommended. So you might want to use multiple content DBs which is easy to configure in Central Administration.

Answer 4

Having folders in document library is irrelevant to performance. It is just for viewing ease.
Sharepoint stores all the documents of a sitecollection in a single database table. Same goes for list items as well.
So splitting lists in folders or multiple lists for size consideration does not make sense.

Check out this msdn article : http://msdn.microsoft.com/en-us/library/dd583133(v=office.11).aspx
Particularly the Content database schema part : "The Docs table stores all the documents of all sites in site collections represented by the database, including, for example, documents in document libraries, attachments, and nodes for each list, but also default.aspx and user pages for each list if they are customized."