0

We are copying more than 1M+ list items with 27 columns from List A to B.

Got to the point where all the items are fetched from List A.

The code throws form digest error after copying around 2K Items.

Code

 function UpdateListItemUsingItemId(A,B,C,D,E,F) {

                var _listItem = {
                    "__metadata": { 'type': 'SP.Data.ABCListNewListItem' },

                    "Project_x0020_Name": A,
                    "IsApproved": B,
                    "IsRejected": C
                };


                $.ajax({
                    url: _spPageContextInfo.webAbsoluteUrl + "/_api/Web/Lists/GetByTitle('ABCListNew')/items",
                    method: 'POST',
                    contentType: "application/json;odata=verbose",
                    async: false,
                    data: JSON.stringify(_listItem),
                    headers: {
                        "accept": "application/json;odata=verbose",
                        "X-RequestDigest": $("#__REQUESTDIGEST").val(),
                        "content-Type": "application/json;odata=verbose"
                        
                    },
                    success: onSuccess,
                    error: onError
                    
                });
                    
        
       
        function onSuccess(data) {
           console.log('List Item Updated');
        }

        function onError(error) {
           console.log(JSON.stringify(error));
        }
      
   };

Modified the code as per post https://stackoverflow.com/questions/32435432/sharepoint-2013-via-rest-api-error-403-forbidden-when-trying-to-create-item

However browsers Chrome and Edge throw Memory Exceeded Error as shown below

enter image description here

Checked the console and got 429 and 500 error as below

enter image description here

Would really appreciate if anyone could let me know if copying over 1M list items is possible? and how do we handle Form Digest expiry on post operation?

Please let me know if any other information is required.

2 Answers 2

1

While you certainly can get a new form digest value by sending a POST to /_api/contextinfo as shown in the other answer you link to, there's another way to update the digest value.

Since you are clearly running this code on a page in SharePoint (because you are using _spPageContextInfo), you can update the form digest for that page by using a built-in SharePoint provided function:

UpdateFormDigest(_spPageContextInfo.webServerRelativeUrl, _spFormDigestRefreshInterval);

That should make subsequent use of $("#__REQUESTDIGEST").val() return a valid digest value.

You could add some code that catches any errors, and if it happens to be a form digest error, then call UpdateFormDigest, and then continue with your copying.


Ok, to address your comments, and after looking again at your code posted above, here's an update with some further suggestions:

I know from the other code that you linked to that you retrieve a bunch of items that you want to copy, and then you loop through them all using jQuery's $.each() function, and in that loop you call your UpdateListItemUsingItemId function. (Which is actually very confusingly named, because in fact you are not using an item's ID to update it, you are creating a whole new list item.)

Looking at the code here:

  1. It looks like you are setting async: false on your $.ajax() call, presumably in order to try and control how the requests are going out since you are looping through a bunch of items using $.each(). This is not good. the first "A" of AJAX stands for asynchronous - it's meant to be asynchronous, so you should learn async programming techniques to handle that kind of code flow, instead of just bypassing it.
  2. It looks like you had to add the contentType: property directly to the AJAX options, because you typo'd the header. When you set the content-type header in the headers object, you should use a lower case t, so it should be "content-type": "app..." not "content-Type": "app...". We can correct that.

Luckily, a for...of loop will respect async / await, so we can reconstruct your loop using modern async programming techniques that simulate synchronous programming, which will make it easier to see where to catch errors and deal with them.

So I would get rid of your $.each() loop and this confusingly named function, and restructure things like this:

// get all the items you want to copy

for (var itemToCopy of allItems) {

    // build your new item payload
    var listItem = {
        // all your fields
        Field1: itemToCopy.Field1,
        Field2: itemToCopy.Field2,
        // etc
    }

    try {
        var newItemResponse = await $.ajax({
            url: _spPageContextInfo.webAbsoluteUrl + "/_api/Web/Lists/GetByTitle('ABCListNew')/items",
            method: 'POST',
            headers: {
                'accept': 'application/json;odata=verbose',
                'content-type': 'application/json;odata=verbose',
                'X-RequestDigest': $('#__REQUESTDIGEST').val()
            },
            data: JSON.stringify(listItem)
        });
        // if no error is thrown, to verify that the
        // new item was created, you _could_:
        //
        // console.log(newItemResponse.d);
        //
        // but that's not really necessary
    }
    catch (err) {
        var isFormDigestError = false;

        // i don't know exactly what you would have to do here,
        // but essentially you will need to figure out how to 
        // check the "err" object to see if it _is_ actually the form digest error

        if (isFormDigestError) {
            UpdateFormDigest(_spPageContextInfo.webServerRelativeUrl, _spFormDigestRefreshInterval);

            // don't forget to re try creating the item
            // that the process failed on
            try {
                var newItemRetry = await $.ajax({
                    url: _spPageContextInfo.webAbsoluteUrl + "/_api/Web/Lists/GetByTitle('ABCListNew')/items",
                    method: 'POST',
                    headers: {
                        'accept': 'application/json;odata=verbose',
                        'content-type': 'application/json;odata=verbose',
                        'X-RequestDigest': $('#__REQUESTDIGEST').val() // getting the _new_ form digest value
                    },
                    data: JSON.stringify(listItem) // but using the _same_ item you built above in this iteration through the loop
                });
            }
            catch (err2) {
                // i really don't know what to tell you if you get here, sorry
            }
        } else {
            // the error is not a form digest error.
            // what do you want to do in this case?
            // it's up to you.  cou could do nothing
            // and keep going through the loop, but i would probably
            throw err;
        }
    }
}
11
  • Thanks Dylan. I added this before the ajax call and its still failing with 403 errors setInterval(function() { UpdateFormDigest(_spPageContextInfo.webServerRelativeUrl, _spFormDigestRefreshInterval); }, 5 * 60000); Can you suggest the best possible way to catch the exception and continue from where we last left?
    – Vivek
    Jul 2, 2021 at 17:21
  • How are you working through your 1M+ items? Are you just running some kind of loop over them and calling UpdateListItemUsingItemId on each one? Or are you using some kind of control / throttling mechanism in your code to make sure each call to update (create?) a list item completes before moving on to the next one? Jul 2, 2021 at 17:30
  • 1
    Wait... are you really using async: false to make your $.ajax calls synchronous??? Jul 2, 2021 at 18:54
  • 1
    And does it work? Does your $.each() loop from the other code "wait" for each call to UpdateListItemUsingItemId to complete before moving on to the next item? Jul 2, 2021 at 19:00
  • 1
    Not sure what you can do about 500 errors, since that is an "internal server" error, but 429 is "Too Many Requests". If you catch one of those errors and determine it is 429, you could check to see if it has a Retry-After header which would tell you how long you need to wait until starting to make requests again. Then... just wait that long, and start the process up again. Jul 5, 2021 at 18:49
0

If you wanted the items to be copied to the new list in the exact same order as the old, it might seem like the best way to do that is synchronously. The options are to use 3rd party tool like Metalogix Essentials, or something like the answers in this question: StackExchange Question. If that isn't a requirement, things are a bit simpler.

For any kind of large volume of moves, it will always be faster to use IDs and for loops. Using the ID in a filter you can process the items in groups of 5000 using REST.

_api/Web/Lists/GetByTitle('yourListNameHere')/Items?$select=yourColumnsNamesHere&$filter=(ID ge 'lowIdhere') and (ID le 'highIdHere')&$top=5000

Some preprocessing would be needed. Get your starting ID (lowest) and your end ID (highest). Have a function handle generating the batch of 5000 IDs. e.g.: 1 to 5000, batch 2 5001 to 10000, etc. [{low:1, high:5000},{low: 5001, high: 10000},{...}]

Take those ID batch starts and end and generate an array of query URLs using the above URL template.

Build out the for loop logic to do the thing you want, in this case copying the item data (with or without file attachments).

Use a Promise.all() to parallel the REST calls (fetch/axios) and return an array of arrays results. Pass the result to your for loop to handle the moves as induvial calls. I've used fetch in the past for this, but I'm sure it can be done with axios just as easily.

There are different ways the above can be built and run, based on the need and performances of the servers and local machine, that would be dependent on your specific situation.

The other answers cover the issues of handling the digest.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.