Improving performance for the Create/Copy files and folders (using code) inside our large document library

Question

We have a SharePoint 2013 on-premises team site and we have a document library which have around 29,000 +++ items, and those files are structured under around 4,000 folders and sub-folder. now inside our Server-side event receiver, i am using the following method to copy files and folder, as follow:-

private void copyfiles(SPItemEventProperties properties, SPListItem currentItem, string from, string to)
        {
    try
     {
      SPFolder folder = properties.Web.GetFolder(properties.Web.ServerRelativeUrl + "/Shared Documents/" + currentItem["ReferenceNumber"] + "/" + from + "/");
      SPDocumentLibrary currentDL = (SPDocumentLibrary)properties.Web.GetList(properties.Web.ServerRelativeUrl + "/Shared Documents/");
      SPListItem softemplete = null;
      TaxonomyFieldValue taxFieldValue = currentItem["CustomerName"] as TaxonomyFieldValue;

      var label = taxFieldValue.Label;
      var titlewithoutspecialchar = currentItem["Title"].ToString().Replace("~", " ").Replace("\"", " ").Replace("#", " ").Replace("%", " ").Replace("&", " ").Replace("*", " ").Replace(":", " ")
              .Replace("<", " ").Replace(">", " ").Replace("?", " ").Replace("/", " ").Replace("\\", " ").Replace("{", " ").Replace("}", " ").Replace("|", " ").Replace(".", " ");
       if (folder.ItemCount > 0)
          {
            SPList list = properties.Web.GetList(properties.Web.ServerRelativeUrl + "/Shared Documents/");
            SPQuery query = new SPQuery();
            query.Folder = folder;
            SPListItemCollection listitem = list.GetItems(query);
              foreach (SPListItem i in listitem)
               {
                 if ((i.Name.ToLower().Contains(currentItem["ReferenceNumber"].ToString().ToLower() + " -") &&
                    (i.Name.ToLower().Contains("sof") || i.Name.ToLower().Contains("pof") || i.Name.ToLower().Contains("qo"))

                     )
                       ||
                     i.Name.ToLower().Contains("request for approval"))
                   {
                     softemplete = i;
                       if (softemplete != null)
                          {
                            byte[] fileBytes = softemplete.File.OpenBinary();
                            string destUrl = properties.Web.ServerRelativeUrl + "/" + currentDL.RootFolder.Url + "/" +
                             currentItem["ReferenceNumber"] + "/" + to + "/" + currentItem["ReferenceNumber"] + " - " + label + " - " + titlewithoutspecialchar.Trim() + ".xlsx";
                             if (i.Name.ToLower().Contains("request for approval.oft"))
                                {
                                    destUrl = properties.Web.ServerRelativeUrl + "/" + currentDL.RootFolder.Url + "/" +
                                   currentItem["ReferenceNumber"] + "/"+to+"/Request for Approval.oft";
                                }
                              else if (i.Name.ToLower().Contains("request for approval.xlsm"))
                                {
                                    destUrl = properties.Web.ServerRelativeUrl + "/" + currentDL.RootFolder.Url + "/" +
                                   currentItem["ReferenceNumber"] + "/" + to + "/Request for Approval.xlsm";
                                }
                                try
                                {
                                    SPFile destFile = currentDL.RootFolder.Files.Add(destUrl, fileBytes, false);
                                }
                                catch
                                {
                                }
                            }

                        }
                    }
                }
            }
            catch
            {
            }


        }

Now the this method is taking quite long to complete the copy (sometimes around 10 seconds), although i am using CAML query to only search the files inside the desired folder usingquery.Folder = folder;. so can anyone provide any advice about improving this method? as i think it is already tuned to the max, is this correct?

Answer 1

Several things to try:

• Instead of foreach loops, switch them out for 'for' loops, they have been proven to be faster: https://www.c-sharpcorner.com/article/for-vs-foreach-in-c-sharp/
• I'd look into concatenation of string more efficiently: https://stackoverflow.com/questions/21078/most-efficient-way-to-concatenate-strings
• I'd replace all instances of properties.Web with a variable at top from SPWeb web = properties.Web; and use web in it's place.
• I'd also replace currentItem["ReferenceNumber"], properties.Web.ServerRelativeUrl, currentDL.RootFolder.Url, and i.Name.toLower() with a variable and use it as a reference. Everytime you call these, it has to fetch to see what the value is.
• I'd also look into using the Contains in the CAML query instead of fetching all items and weeding the items out in memory. See here: caml query for text field(Title field) contains username in SharePoint hosted app

Answer 2

all that was mentioned by Mike are valid tips and should be deferentially checked and considered

other thing I would check which may have a huge performance impact is the way You query items from SharePoint. Using SPQuery and GetItems() method is not the best way. First of all it does not provide any caching and returns SPListItem objects that have all metadata (columns) loaded (which sometimes is not the best option as You for example would like to only use information from 4 or 5 columns)

better option is to use CrossListQueryInfo with cache turned on. The difference is that it will return a DataTable by default. The advantage is possible caching and also it is mandatory to provide the list of fields You plan to be using (and only those will be initialized in the returned list of objects). Example implementation of this kind of query would look like:

    DataTable dt = null;
    CrossListQueryInfo crossListQueryInfo = new CrossListQueryInfo();
    crossListQueryInfo.Query = "<Where><IsNotNull><FieldRef Name='Title'/></IsNotNull></Where>";
    crossListQueryInfo.ViewFields = "<FieldRef Name=\"Title\" />";
    crossListQueryInfo.Lists = "<Lists><List ID='[GIUD OF LIST]'/></Lists>";
    crossListQueryInfo.Webs = "<Webs Scope='Recursive' />";
    crossListQueryInfo.UseCache = true;
    CrossListQueryCache crossListQueryCache = new CrossListQueryCache(crossListQueryInfo);
    dt = crossListQueryCache.GetSiteData(SPContext.Current.Site);
    return dt;

Please refer to this article to check different methods of query and their performance. Check this chart... the CorssListQueryInfo is the blue line at the bottom of the chart :).

Hope this will be of any help :). I know SP may be performance tricky :)