Best practice to bulk upload records to an existing list

Question

I need to do a bulk upload over 10k records to an existing SharePoint list. I am using MS access and it is very slow and keep crashing.

Is any better way of doing it?

Thanks

Answer 1

You can either use powershell or create a simple console application to read from the source and write to SharePoint list. MS Access would find it difficult to cope with such volume of data..

Here's a simple c# code to create console app

static void ImportList() {

            // update site url here
            string siteURL = "http://xxxxxxxxxxxxx";

            SPSecurity.RunWithElevatedPrivileges(delegate () { 

            using (SPSite site = new SPSite(siteURL)) {
                using (SPWeb web = site.OpenWeb()) {

                        SPList listTo = web.Lists[new Guid("id")]; // new list
                        SPList listFrom = web.Lists[new Guid("id")]; // source list

                        // update your query 
                        var qLogs = new SPQuery()
                        {
                            Query = @"<Where><Neq><FieldRef Name='Title' /><Value Type='Text'></Value></Neq></Where>"
                        };

                        var rLogs = listFrom.GetItems(qLogs);

                        foreach (SPItem item in rLogs)
                        {
                            // match your field items 
                            SPListItem discussionItem = listTo.AddItem();
                            discussionItem["Title"] = item["Title"];
                            discussionItem["Created"] = item["Created"];
                            discussionItem["Modified"] = item["Modified"];
                            discussionItem["Author"] = item["Author"];
                            discussionItem["Editor"] = item["Editor"];

                            discussionItem.Update();

                            Console.WriteLine("testing........");
                            Console.WriteLine("Title: " + item["Title"]);                                
                            Console.WriteLine("Created: " + item["Created"]);
                            Console.WriteLine("Modified: " + item["Modified"]);
                            Console.WriteLine("Author: " + item["Author"]);
                            Console.WriteLine("Editor: " + item["Editor"]);
                        }


                    }
            }

            }); // end run with elevated privilages
        }

Also please note that there's 5000 item limitation in SharePoint lists.

Answer 2

1. For existing SharePoint list:

If the data is in excel format.

You can create a utility in CSOM to copy the data from excel and insert into existing SharePoint list.

2. To create new list with same name:

Delete the existing list and copy the name of existing list. Import the data from Access to SharePoint list by creating a new list with same name. Reference link