Can I have a library with more than 5000 items in it?

Question

I have created this document library for my organization where there will be a huge number of documents that will be uploaded. I am aware that this thresh hold limit can be increased from central admin but I have a few concerns.

1. The library has heavily nested folder structures
2. We can't have separate libraries as a part of Business requirement
3. It doesn't have any lookup, group by or sum field, nor any filtered views.
4. No bulk update/edit of all the document's metadata is expected to be performed.

It just has a lot of documents stored in their respective folders and the number of folders+documents will very soon exceed 5000. Also each page view has a max of 150 items in it.

So I just want to know, what will be the impact when the thresh hold is crossed? What are the possible impacts if I increase the thresh hold to 10000, like what functionalities are likely to disrupt?

This is a very critical scenario for me as the thresh hold will be reached within a couple of hours time and any help/suggestion is highly appreciated.

Answer 1

From experience :

As stated by others, you can have a view that returns more than the threshold (default 5000).

There are three things that can be used to achieve this :

1. Paging. Flat view with no filer/grouping/total.
2. Filter/Grouping/Total, but you cannot use one of those on columns that are not indexed. If you need to do so, do it before reaching the threshold because the indexing mechanism will prevent it.
3. Metadata Navigation. You will be able to show more than the threshold, but only a limited list will be returned until you filter your data.

Notes :

• the list threshold can be changed in central admin, but I don't recommend this approach as it can impact performance.
• WFE server admins (not just site collection or web application admins) have a different limit.
• threshold can be programmatically changed (disabled) for a particular list (by farm admin via PowerShell or developer via code).

Answer 2

Posted comment answering the question

The limitation is on page views. 1 page view can max contain 5000 items. If you have 150 items per page view, you should be fine.

Answer 3

Actually, it doesn't matter that much if the end result is one item, a dozen or 4999. The SQL database must have fewer than 5000 items in the subset to be returned by the first filtering operation in the query to avoid table lock escalation.

In your case, if the requirements don't change, you're fine, because the database shall filter documents by folder path and your 150 documents limit is in the green zone.

If you need more information, here goes:

Therefore, the first filtering operation in the CAML <Query><Where><!--<And></And>--></Where></Query> must use an indexed column that will surely filter a narrow enough set of results for other refining criteria. A sample of that is ID in any list or a Parent workflow instance in Workflow History lists which usually grow far beyond 5000 items but still work in out-of-the-box scenarios. Paging also depends on the view being sorted on an indexed column.

Hint. Metadata navigation and filtering has a feature that lets you enable automatic creation of column indices.

Operations such as <Eq/>, <In/>, <Gt/>, <Lt/>, <Geq/>, <Leq/>, <IsNull/>, <IsNotNull/> are good enough for such narrowing on indexed values.

Please try to avoid <Or/>, <Neq/> and <Contains/> in large libraries and lists, at least at the top level of the query, and be aware that views without folders (<View Scope='RecursiveAll'/>) also tend to be affected by this limitation. You can work around the use of Or, Neq and Contains by adding calculated columns that ensure that the specific condition is met in the given item and may be queried by more efficient comparison operations.

Helpful links:

• MSDN: Handling Large Folders and Lists
• SharePoint lists training, part V: Techniques for managing large lists
• Designing large lists and maximizing list performance (SharePoint Server 2010)

Answer 4

In past I notice when a client had more then 5000 documents with many attachments
1. The performance was so slow
2. It was hard to navigate the right document for users.
3. For some reason the MS SQL Index was broken and items were not coming up search terms. it was just like asking for trouble.

Its an issue your better of splitting it into folders or library. if you have large view items.