0

I have a requirement to create 300,000 folders in SharePoint document library, so for that I have developed an console application which will create the folders in batch operations.

The below code which I have written for the same will do as follows:

  1. First take the 1000 folder named which needs to be created
  2. Loop through the above folder list and create a batch of 100 records
  3. Update the item for the first 99 records and after 100, call the ExecuteQuery

    public static void CreateFolderInBatch(List<string> folderNames, string listName, string spSiteURL, NetworkCredential credential)
    {
        try
        {
            using (ClientContext clientContext = new ClientContext(spSiteURL))
            {
                //Giving required Credentials
                clientContext.Credentials = credential;
    
                //Get the list by title
                Microsoft.SharePoint.Client.List destinationLibrary = clientContext.Web.Lists.GetByTitle(listName);
    
                FolderCollection folderCollection = destinationLibrary.RootFolder.Folders;
                clientContext.Load(folderCollection);
                clientContext.ExecuteQuery();
    
                //Get total records count
                int totalRecords = folderNames.Count;
    
                int remainingCounter = totalRecords;
                int mainCounter = 0;
    
                //Loop through all the records
                //Create a batch of 100 records and create folder on SharePoint Document Library
                while (mainCounter < totalRecords)
                {
                    int loopCounter = 0;
    
                    //Check if the remaining records exceeds 100
                    if (remainingCounter >= 100)
                    {
                        loopCounter = 100;
                    }
                    else
                    {
                        loopCounter = remainingCounter;
                    }
    
                    //calculate the remaining no of records
                    remainingCounter = (totalRecords - mainCounter) - loopCounter;
    
                    //Loop through records 100 times
                    for (int i = 0; i < loopCounter; i++)
                    {
                        //Get the folder name
                        string folderName = folderNames[mainCounter];
                        bool folderExists = false;
    
                        Folder sharePointFolder = folderCollection.FirstOrDefault(x => x.Name == folderName);
    
                        if (sharePointFolder != null)
                        {
                            folderExists = true;
                        }
    
                        //If the flag is false, then only create folder
                        if (folderExists == false)
                        {
                            //Call function to create a batch of 100 records
                            CreateSharePointFolder(clientContext, destinationLibrary, folderName);
                        }
    
                        mainCounter++;
                    }
    
                    clientContext.ExecuteQuery();
                }
            }
        }
        catch (Exception ex)
        {
            throw new Exception("CreateFolder: " + ex.ToString());
        }
    }
    
    public static void CreateSharePointFolder(ClientContext clientContext, Microsoft.SharePoint.Client.List list, string Foldername)
    {
        try
        {
            //Create new list item object, set item name and update the list
            ListItemCreationInformation newItemInfo = new ListItemCreationInformation();
            newItemInfo.UnderlyingObjectType = FileSystemObjectType.Folder;
            newItemInfo.LeafName = Foldername;
            Microsoft.SharePoint.Client.ListItem newListItem = list.AddItem(newItemInfo);
            newListItem.Update();
        }
        catch (Exception ex)
        {
            throw new Exception("CreateSharePointFolder: " + ex.ToString());
        }
    }
    

Now when I run the console application, the first 1000 folders have been created successfully.

But after that I am getting the below error

   CreateFolder: Microsoft.SharePoint.Client.ServerException: The operation has timed out.
   at Microsoft.SharePoint.Client.ClientRequest.ProcessResponseStream(Stream responseStream)
   at Microsoft.SharePoint.Client.ClientRequest.ProcessResponse()
   at Microsoft.SharePoint.Client.ClientRequest.ExecuteQueryToServer(ChunkStringBuilder sb)
   at Microsoft.SharePoint.Client.ClientRequest.ExecuteQuery()
   at Microsoft.SharePoint.Client.ClientRuntimeContext.ExecuteQuery()
   at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()
   at CMG.Kizuna.CRM.CreateDocumentLocation.Utility.CreateFolderInBatch(List`1 folderNames, String listName, String spSiteURL, NetworkCredential credential)

Can anyone please help me in order to resolve this issue?

Thanks in advance.

3

You can increase client context request timeout. please use below code for same.

clientContext.RequestTimeout = -1;

Hope it will work for you.

  • Thanks for your answer. Let me try and I will get back to you. – User5590 Feb 15 '17 at 6:59
  • It helped a little bit. I have set it the timeout to -1 and test the console application. The application worked upto 8000 folder creation and then again threw the timeout error. – User5590 Feb 15 '17 at 9:28
1

You can use PnP's Core throttling sample in your code as below:

try{
    //Giving required Credentials
    clientContext.Credentials = credential;

    //Get the list by title
    Microsoft.SharePoint.Client.List destinationLibrary = clientContext.Web.Lists.GetByTitle(listName);

    FolderCollection folderCollection = destinationLibrary.RootFolder.Folders;
    clientContext.Load(folderCollection);
    // Extension method for executing query with throttling checks
    ctx.ExecuteQueryWithIncrementalRetry(5, 30000); //5 retries, with a base delay of 10 secs.

.... further code

}
catch(MaximumRetryAttemptedException mex)
{

}

The ExecuteQueryWithIncrementalRetry is an extension method, the implementation of that is as below:

// This is the extension method. 
// The first parameter takes the "this" modifier
// and specifies the type for which the method is defined. 
/// <summary>
/// Extension method to invoke execute query with retry and incremental back off.
/// </summary>
/// <param name="context"></param>
/// <param name="retryCount">Maximum amount of retries before giving up.</param>
/// <param name="delay">Initial delay in milliseconds.</param>
public static void ExecuteQueryWithIncrementalRetry(this ClientContext context, int retryCount, int delay)
{
    int retryAttempts = 0;
    int backoffInterval = delay;
    if (retryCount <= 0)
        throw new ArgumentException("Provide a retry count greater than zero.");

    if (delay <= 0)
        throw new ArgumentException("Provide a delay greater than zero.");

    // Do while retry attempt is less than retry count
    while (retryAttempts < retryCount)
    {
        try
        {
            context.ExecuteQuery();
            return;

        }
        catch (WebException wex)
        {
            var response = wex.Response as HttpWebResponse;
            // Check if request was throttled - http status code 429
            // Check is request failed due to server unavailable - http status code 503
            if (response != null && (response.StatusCode == (HttpStatusCode)429 || response.StatusCode == (HttpStatusCode)503))
            {
                // Output status to console. Should be changed as Debug.WriteLine for production usage.
                Console.WriteLine(string.Format("CSOM request frequency exceeded usage limits. Sleeping for {0} seconds before retrying.", 
                                backoffInterval));

                //Add delay for retry
                System.Threading.Thread.Sleep(backoffInterval);

                //Add to retry count and increase delay.
                retryAttempts++;
                backoffInterval = backoffInterval * 2;
            }
            else
            {
                throw;
            }
        }
    }
    throw new MaximumRetryAttemptedException(string.Format("Maximum retry attempts {0}, has be attempted.", retryCount));
}

Reference implementation -PnP Core.Throttling

  • Thanks for your valuable answer, will check and update you on this. – User5590 Feb 15 '17 at 8:15
  • did this work for you @User5590? – Colbs Dec 11 '17 at 22:39
  • We've recently hit Microsoft's new throttling limits with all our CSOM console apps that used to work without any issue.I've implemented – Panoone Feb 22 '18 at 3:36
  • Premature send above. :\ We've recently hit Microsoft's new throttling limits with all our CSOM console apps that used to work without any issue. I implemented the ExecuteQueryWithIncrementalRetry extension method which MS is now pushing as best practice and am now continually receiving the dreaded "The property or field 'FieldName' has not been initialized..." exception after every query execution. Putting in break points and additional delays mostly solves this but also makes the code as slow as performing the process manually. – Panoone Feb 22 '18 at 3:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.