2

I have a CSOM desktop application adding huge data in the list from a excel sheet. Its is taking 5 to 6 hours to upload 150000 entries(Not files). Does it take so long or can we improve performance.

Restriction is we have to do it Desktop application only.

Thanks, Divakar

Improve this question
4
  • 2
    can you share code snippet of how you are doing it currently ?
    – Gautam Sheth
    Sep 10, 2018 at 7:09
  • 1
    As correctly said by Gautam, if possible to share your code then we can evaluate to check if the code is optimized enough to perform this huge update. I remember we had done a similar project where we updated 65000 items in SharePoint list through CSOM and that took about 90 minutes
    – Zakir HC
    Sep 10, 2018 at 7:39
  • 1
    What SP version?
    – M.P.
    Sep 10, 2018 at 15:03
  • yes, What SharePoint version is that?
    – Denis Molodtsov
    Sep 18, 2018 at 21:01

3 Answers 3

Reset to default
0

You can use multi-threading to improve performance of your code.This gets multiple file upload calls on different thread and sends multiple request to server for upload.

For reference

Improve this answer
2
  • 1
    This is a bit far fetched. You are assuming the bottleneck is CPU load on the client. This answer should be a comment.
    – helb
    Sep 10, 2018 at 9:38
  • 1
    @helb would be nice to compare batching with ExecuteQuery (100-200 items at a time) + multi threading. I think, I've seen something similar with REST here: youtube.com/watch?v=b7AywSeYVhI&t=24s (don't have time to watch it right now) I predict that "batching" multiple item creation in a single ExecuteQuery + miltithreading should be much faster than doing it in one thread. If I have time - I will create a test tonight
    – Denis Molodtsov
    Sep 17, 2018 at 14:27
0

While Inserting or Updating the list about 150000 item it Takes long time due to RequestTimeout. To increase the performance Set The RequestTimeout to Standard time or else use Multi Threading. Here the Below link for inserting items through MultiThreading.

Visit:- http://www.sharepointpals.com/post/Implementing-Multi-Threading-with-MaxDegreeOfParallelism-in-SharePoint-using-CSOM-An-Insight

Improve this answer
0

First of...

I most likely don't like anyone uploading that many items into one list. Make sure you have your list indices set correctly before you start this.

You can most likely achieve some improvement if you try to batch several item creations into single calls. Do this by only calling the ExecuteQuery method after a few items (200 for example) have been inserted.

I ran a small test with CSOM against my demo SPO.

I was able to create ~300 items per patch. The duration for each batch was between 22 and 31 seconds (List currently has about 11K items and times are still stable)

With 150000 rows this would still be 15000 seconds or slightly over 4 hours.

When I tried to create 1000 at once I got a timeout error since I didn't increase the request timeout.

Improve this answer
4
  • I didn't have time to actually confirm it, but indexing should incur a performance overhead. Maintaining the index adds processing to creating, updating, or deleting items from a list. I remember I had a list of 600K items where it took us 30 seconds to delete a single item. I guess this is because every time the entire index had to be rebuilt. So I would probably suggest setting up indexes after adding a ton of items to the list not before
    – Denis Molodtsov
    Sep 17, 2018 at 14:32
  • Yes... But you wont be able to add the indexes you need after the items are added. Since this will trigger the 5000 item threshold, and you cant create the index unless you disable this option (Which I don't like and which is not possible in SPO). So adding the indices afterwards might not be possible, depending on where you are
    – Heiko Hatzfeld_MSFT
    Sep 18, 2018 at 8:08
  • Yea, I've heard about this. I don't remember if I faced this issue. I have just created an index in a list with 8.000 items with no issues. If it's an issue and If it's not SharePoint Online - we can temporarily increase the threshold to make sure indexes cannot be created. Quite honestly, it's strange that we can hit threshold when creating an index. Because index is rebuilt every time you update a list item any way. Can you create a large list and create an index for it? I wonder if anyone can replicate this theory. Maybe it was true for some old SP versions some time ago?
    – Denis Molodtsov
    Sep 18, 2018 at 21:09
  • I have found this discussion: sharepoint.stackexchange.com/questions/192939/… it seems people indeed had issues with not being able to create indexes. I'm trying to do it in SPO and it works like a charm
    – Denis Molodtsov
    Sep 18, 2018 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.