Import bulk data (over 5000 List Items) into SharePoint

Question

I have an Microsoft Office 2010 AddIn which imports Data into SharePoint. The tool creates a Dictionary with over 5000 Values. Every Value is a C#-Class with Data and represents a SharePoint List Item. Because there are lookup relationships between almost all Dictionary-Values (ListItems) i use two loops:

• in the first loop I create the ListItems and write back the generated ListItemIDs into the Dictionary
• in the second loop I already have IDs for all ListItems and update the Lookups for all theese created ListItems

Is there a way to minimize the number of my queries ( in worst case scenario there are 5000 ExecuteQuery() for the first loop + 5000 for the second).

Answer 1

A little-known and definitely under-used method is SPWeb.ProcessBatchData. Use this to really effectively perform operations in bulk - I still think there's a limit to what can be done at once, but depending on the schema of your lists this could still be the most efficient way (most definitely in terms of requests sent).