Iterate over all documents in a site

Question

I'm trying to iterate recursively over all documents in a given site. This is what I'm using:

    using (System.IO.StreamWriter file = new System.IO.StreamWriter("docs.txt"))
    {
        foreach (var lib in docLibs)
        {
            CamlQuery query = new CamlQuery();
            query.ViewXml = "<View Scope = 'RecursiveAll'></View>";
            ListItemCollection items = lib.GetItems(query);
            clientContext.Load(items);
            clientContext.ExecuteQuery(); // Exception thrown here
            foreach (var item in items)
            {
                Microsoft.SharePoint.Client.File f = item.File;
                if (f.ServerObjectIsNull.HasValue && !(bool)f.ServerObjectIsNull)
                    file.WriteLine(f.Name);
            }
        }
    }

The error I'm getting is "The request uses too many resources" I'm guessing some List in my site has too many documents and therefore one of the requests is failing. Is there anyway to retrieve the documents chunk by chunk or some other approach to retrieve all documents?

Answer 1

Using CSOM you can add a min and max ID into your query to restrict your query when iterating through a large query.for example (where ID > 1 and ID < 5000).

When using the SharePoint Server api, you can use theContentIterator, to help with accessing more than 5,000 items in a large list without hitting a list throttling limit and receiving an SPQueryThrottleException.

ContentIterator implements a callback pattern for segmenting the query for processing a single item at a time. Consider using this capability if you need to process a large number of items that may exceed a throttling limit.

MSDN shows the following example

static int exceptions = 0;
static int items = 0;

protected void OnTestContentIterator(object sender, EventArgs args)
{
    items = 0;
    exceptions = 0;
    string query1 = @"<View>
        <Query>
            <Where>
                <And>
                    <BeginsWith>
                        <FieldRef Name='SKU' />
                        <Value Type='Text'>S</Value>
                    </BeginsWith>
                </And>
            </Where>
        </Query>
    </View>";

    ContentIterator iterator = new ContentIterator();
    SPQuery listQuery = new SPQuery();
    listQuery.Query = query1;
    SPList list = SPContext.Current.Web.Lists["Parts"];
    iterator.ProcessListItems(list,
        listQuery,
        ProcessItem,
        ProcessError
    );
}

public    bool ProcessError(SPListItem item, Exception e) 
{ 
    // process the error
    exceptions++; 
    return true; 
}
public void ProcessItem(SPListItem item)
{
    items++;
    //process the item.
}

Answer 2

Please try the below link ; Working with Large Lists

or Try optimizing the query by specifying only the fields you need to check and also try pagination when querying the list as shown in the below link .Pagination in Large Lists