1

I have a CSOM desktop application adding huge data in the list from a excel sheet. Its is taking 5 to 6 hours to upload 150000 entries(Not files). Does it take so long or can we improve performance.

Restriction is we have to do it Desktop application only.

Thanks, Divakar

  • 2
    can you share code snippet of how you are doing it currently ? – Gautam Sheth Sep 10 '18 at 7:09
  • 1
    As correctly said by Gautam, if possible to share your code then we can evaluate to check if the code is optimized enough to perform this huge update. I remember we had done a similar project where we updated 65000 items in SharePoint list through CSOM and that took about 90 minutes – Zakir HC Sep 10 '18 at 7:39
  • 1
    What SP version? – M.P. Sep 10 '18 at 15:03
  • yes, What SharePoint version is that? – Denis Molodtsov Sep 18 '18 at 21:01
0

You can use multi-threading to improve performance of your code.This gets multiple file upload calls on different thread and sends multiple request to server for upload.

For reference

  • 1
    This is a bit far fetched. You are assuming the bottleneck is CPU load on the client. This answer should be a comment. – helb Sep 10 '18 at 9:38
  • 1
    @helb would be nice to compare batching with ExecuteQuery (100-200 items at a time) + multi threading. I think, I've seen something similar with REST here: youtube.com/watch?v=b7AywSeYVhI&t=24s (don't have time to watch it right now) I predict that "batching" multiple item creation in a single ExecuteQuery + miltithreading should be much faster than doing it in one thread. If I have time - I will create a test tonight – Denis Molodtsov Sep 17 '18 at 14:27
0

While Inserting or Updating the list about 150000 item it Takes long time due to RequestTimeout. To increase the performance Set The RequestTimeout to Standard time or else use Multi Threading. Here the Below link for inserting items through MultiThreading.

Visit:- http://www.sharepointpals.com/post/Implementing-Multi-Threading-with-MaxDegreeOfParallelism-in-SharePoint-using-CSOM-An-Insight

0

First of...

I most likely don't like anyone uploading that many items into one list. Make sure you have your list indices set correctly before you start this.

You can most likely achieve some improvement if you try to batch several item creations into single calls. Do this by only calling the ExecuteQuery method after a few items (200 for example) have been inserted.

I ran a small test with CSOM against my demo SPO.

I was able to create ~300 items per patch. The duration for each batch was between 22 and 31 seconds (List currently has about 11K items and times are still stable)

With 150000 rows this would still be 15000 seconds or slightly over 4 hours.

When I tried to create 1000 at once I got a timeout error since I didn't increase the request timeout.

  • I didn't have time to actually confirm it, but indexing should incur a performance overhead. Maintaining the index adds processing to creating, updating, or deleting items from a list. I remember I had a list of 600K items where it took us 30 seconds to delete a single item. I guess this is because every time the entire index had to be rebuilt. So I would probably suggest setting up indexes after adding a ton of items to the list not before – Denis Molodtsov Sep 17 '18 at 14:32
  • Yes... But you wont be able to add the indexes you need after the items are added. Since this will trigger the 5000 item threshold, and you cant create the index unless you disable this option (Which I don't like and which is not possible in SPO). So adding the indices afterwards might not be possible, depending on where you are – Heiko Hatzfeld_MSFT Sep 18 '18 at 8:08
  • Yea, I've heard about this. I don't remember if I faced this issue. I have just created an index in a list with 8.000 items with no issues. If it's an issue and If it's not SharePoint Online - we can temporarily increase the threshold to make sure indexes cannot be created. Quite honestly, it's strange that we can hit threshold when creating an index. Because index is rebuilt every time you update a list item any way. Can you create a large list and create an index for it? I wonder if anyone can replicate this theory. Maybe it was true for some old SP versions some time ago? – Denis Molodtsov Sep 18 '18 at 21:09
  • I have found this discussion: sharepoint.stackexchange.com/questions/192939/… it seems people indeed had issues with not being able to create indexes. I'm trying to do it in SPO and it works like a charm – Denis Molodtsov Sep 18 '18 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.