Cannot Query Document Library with > 5K folders

Question

I have some SP .NET Object library code that will allow me to query for the contents of a specific folder in a document library. It worked fine in a UAT environment where there were only a few dozen folders, but when I moved it to Production where there were 14K folders, it won't return the folders anymore.

I assume that I need a way to get directly to the folder, instead of retrieving the list of folders and then getting the one I need, but I don't know how to do it? I think i could do it with a CAML query, but I can't find a way to get directly to the folder and get the list of files?

Assumumptions:
Site Name = https://xyz.sharepoint.com/sites/CRMDocs/

Library Friendly Name = Case

Library Path = incident

Sample Folder = 0312380e

Full Path = https://xyz.sharepoint.com/sites/CRMDocs/incident/0312380e

Web web = clientContext.Web;
ListCollection lists = web.Lists;

var docLib = web.Lists.GetByTitle(entityFriendlyName);
clientContext.Load(docLib, d => d.Title, d => d.RootFolder.Name);
clientContext.ExecuteQuery();

String folderUrl = String.Format("/{0}/{1}", docLib.RootFolder.Name, folderName);

var folders = docLib.RootFolder.Folders;
clientContext.Load(folders, fldrs => fldrs.Include(fldr => fldr.ServerRelativeUrl));
clientContext.ExecuteQuery();

// When there are more than 5K folders, folders collection is NULL
var folder = folders.FirstOrDefault(f => f.ServerRelativeUrl.ToLower().EndsWith(folderUrl.ToLower()));
if (folder != null)
{
    var files = folder.Files;

    clientContext.Load(files);
    clientContext.ExecuteQuery();
    // Process Files
}

Answer 1

In SharePoint Online, your only choice is to use REST. CSOM will not work. You call the REST endpoint and it will return a batch of 100 or 200 items, and if more exist, will return a paged link to access the next batch of data.

I've used this technique in the past to update 100k+ items in a document library with success (in PowerShell), CSOM would not work. Marc Anderson has it detailed here, http://sympmarc.com/2016/11/30/25696.

This is a sample from when I used it in PowerShell:

#https://www.itunity.com/article/working-lists-list-items-sharepoint-rest-service-windows-powershell-2077

function Get-Items{
    Param(
        [Parameter(Mandatory=$true,Position=0)][ValidateNotNull()]
        [string]$Url
    )
    $list = Invoke-SPORestMethod -Url "https://tenant.sharepoint.com/sites/site/_api/Web/Lists/getbytitle('Documents')" 
    $itemType = $list.ListItemEntityTypeFullName 
    $digest = (Invoke-SPORestMethod -Url "https://tenant.sharepoint.com/sites/site/_api/contextinfo" -Method "POST").GetContextWebInformation.FormDigestValue 
    $listItems = Invoke-SPORestMethod -Url $Url
    Write-Host "Processing batch starting at "$listItems.results[0].Id -ForegroundColor Yellow

    foreach($li in $listItems.results){
        $newTitle = $li.FileLeafRef.ToString()
        if($li.Title -ne $newTitle){
            $id = $li.Id
            $pgurl = "https://tenant.sharepoint.com/sites/site/_api/Web/Lists/getbytitle('Documents')/items($id)"
            Write-Host "Updating $pgurl" -ForegroundColor White
            $metadata = "{ '__metadata': { 'type': '$itemType' }, 'Title': '$newTitle'}"
            $item = Invoke-SPORestMethod -Url $pgurl -Method "POST" -XHTTPMethod "MERGE" -Metadata $metadata -RequestDigest $digest -ETag "*"
        }
    }
    Write-host "Batch complete" -ForegroundColor Green
    if($listItems.__next){
        Get-Items -Url $listItems.__next    
    }
}
cls
Get-Items -Url "https://tenant.sharepoint.com/sites/site/_api/Web/Lists/getbytitle('Documents')/items?`$select=FileLeafRef,Title,Id&`$orderby=Id%20desc&`$filter=(ID%20le%20313031)"

Answer 2

Not a specific answer, but I did finally find a c# set of functions that will allow you to connect to SPOnline and pass in a REST call.

http://journeyintocrm.com/archives/1152

Answer 3

Actually, after working with this some more, I found an even better way that does not require a REST call.

Once you have the SharePoint clientcontext.Web, make a call to web.GetFolderByServerRelativeUrl(folderNameString);

Web web = clientContext.Web;
ListCollection lists = web.Lists;

var docLib = web.Lists.GetByTitle(entityFriendlyName);
clientContext.Load(docLib, d => d.Title, d => d.RootFolder.Name);
clientContext.ExecuteQuery();

String folderUrl = String.Format("/{0}/{1}", docLib.RootFolder.Name, folderName);

Folder folder = web.GetFolderByServerRelativeUrl(folderUrl);

if (folder != null)
{
    var files = folder.Files;

    clientContext.Load(files);
    clientContext.ExecuteQuery();
    // Process Files
}