2

I need to create about 8000 sites in a site collection, based on Title and path attributes from a database table with 8000 records. I wrote a simple console based application in C# that utilizes the Server Object Model and the following algorithm:

  1. Connect to this database and retrieve all the rows in this table
  2. Connect to the desired SharePoint Site Collection (SPSite)
  3. For each row in rows, Add a website under SPSite.RootWeb.Webs using Add method

This application is running on one of the web servers, not sure if that matters. Actual code for child site creation:

/*----
 * Create the child web in the site collection.
 *----*/
using (SPSite siteCollection = new SPSite(siteCollectionUrl))
{
    using (SPWeb rootWeb = siteCollection.RootWeb)
    {
        // Create a new child web in the site collection using the input parameters defining the child web.
        SPWeb newWeb = rootWeb.Webs.Add(webUrl, title, description, 1033, webTemplate, useUniquePermissions, false);
        newWeb.Dispose();
     }
 }

The setup is a medium SharePoint 2016 farm with two app servers (MinRole: App + Search) and two web servers (MinRole: Web + Distributed Cache).

The application is still running. At first, creation of the child site was taking about 15 seconds. After about 500 sites, that same operation is taking 40 seconds. Now, at about 1500 sites, site creation is taking 70 seconds. If this pattern keeps growing, I am afraid it will take way too long to create all 8000 sites. What may be the reason for this slow down? Is there any way to optimize and increase the speed of site creation, hopefully dramatically? Because I have to provision several more site collections that are similarly large.

Update 1/20/2018: I re-attempted the same site collection provisioning with a PowerShell based script:

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction Stop
$ErrorActionPreference = "Stop"

$LogFile = "C:\Development\TestLargeSiteCollectionCreation_$(Get-Date -Format "yyyyMMdd").log"

Function WriteToLog
{
  #[CmdletBinding()]
  Param([string] $logdata)

  Write-Host $logdata; Out-File -filepath $LogFile -inputobject $logdata -Append
}

# Create site collection
$SiteCollUrl = "http://<portal-url>/alsc"
$SiteCollTitle = "A Large Site Collection"
$SiteOwner = "CONTOSO\spadmin"
$WebTemplate = Get-SPWebTemplate "STS#0"

$SiteColl = Get-SPSite -Identity $SiteCollUrl -ErrorAction SilentlyContinue
if (!$SiteColl) {
    $result = New-SPSite -Name $SiteCollTitle -Url $SiteCollUrl -Template $WebTemplate -OwnerAlias $SiteOwner
    WriteToLog "Created site => $result"
}
else {
    WriteToLog "Found site => $SiteColl"
}

Get-Content "C:\Development\childSiteList.txt" | ForEach-Object {
    $childpath = $_

    $c = Start-SPAssignment
    $mc = Measure-Command { $newChildWeb = New-SPWeb -Name $childpath -Url "$SiteCollUrl/$childpath" -Template $WebTemplate -UniquePermissions -UseParentTopNav -AssignmentCollection $c -ErrorAction SilentlyContinue }
    if ($newChildWeb) {
        WriteToLog "Created case site $SiteCollUrl/$childpath in $($mc.Seconds) second(s)"
    }
    else {
        WriteToLog "Unable to create case site $SiteCollUrl/$childpath"
    }
    Stop-SPAssignment $c
}

I was previously told by a MS SharePoint Support engineer that SharePoint PowerShell cmdlets are preferred to SharePoint Object Model for this type of provisioning, which prompted me to try the above approach. The script ran marginally faster, but I observed that the rate at the beginning was 11 seconds per child site and by the time the script ended, the rate averaged at 40 seconds.

Does a moderately sized farm comprising of 2 servers with MinRole of Web Front End plus Distributed Cache and 2 other servers with MinRole of Application plus Search and finally a 2-server SQL Server cluster somehow have anything to do with this dismal performance? I have a staging environment with a similar set up but comprising of one server each, that is significantly faster and the rate is uniformly set at 10-11 seconds per site.

Second Update 1/21/2018: Corresponding with the creation of each subsite, I see the following error in Event Viewer (TestLargeSiteCollectionCreation is the name of the C# console application I wrote to provision a large number of sub-sites in an existing site collection):

A failure was reported when trying to invoke a service application:

EndpointFailure

Process Name: TestLargeSiteCollectionCreation

Process ID: 6428

AppDomain Name: TestLargeSiteCollectionCreation.exe

AppDomain ID: 1

Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:d623596b16ff4873bac023833775c635#authority=urn:uuid:92c677282f3f411185886510365a07db&authority=https:TestLargeSiteCollectionCreationFTestLargeSiteCollectionCreationF:32844TestLargeSiteCollectionCreationFTopologyTestLargeSiteCollectionCreationFtopology.svc

Active Endpoints: 3

Failed Endpoints:1

Affected Endpoint: http://:32843/d623596b16ff4873bac023833775c635/ProfileService.svc

As I mentioned previously, the production environment has 4 MinRole servers - App01/02 are App+Search, Web01/02 are Web+Distributed-Cache.

  • 1
    Is the usage of the memory growing on the console app? Maybe there is some issue with memory leak (although your code from the outset looks ok) – Marek Sarad Jan 17 '18 at 16:08
  • 3
    Just to add to this, consider your future upgrades - a significant number of subsites will cause you to have significant upgrade times if you go the route of a database attach upgrade (on the order of 1+ days...). In addition, Microsoft is moving towards a model of having flat hierarchy for sites, that is, not using subsites. – Trevor Seward Jan 17 '18 at 16:19
  • 4
    Sounds like these are all going into a single content database. I'd recommend changing your auto growth settings so that the DB doesn't have to keep reallocating. technet.microsoft.com/en-us/library/… – theChrisKent Jan 17 '18 at 16:23
  • @MarekSarad the memory of the console app grew to about 3.5 GB quite soon and did not change much after that. I would almost say that is not an issue since I have use the same application in MOSS 2007 (in the past) without much trouble. – Web User Jan 17 '18 at 18:03
  • 1
    @Web User this is a known problem with PowerShell. Once it consumes over 1 to 1.5GB of memory it starts slowing down. The usual workaround is to launch each action, in this case the site create, in a separate process using Start-Job. Once PowerShell consumes that much memory the session will need to be closed and a new session started. – Underverse Jan 24 '18 at 3:12
3
+100

I can't tell from your code snippets, but are you creating the SPSite and the SPWeb in each iteration? If so, those are both "non managed memory" objects and will create a lot of objects to be cleaned up by the GC in the background.

Are you using this:

foreach (row in rows) 
{
  using (SPSite siteCollection = new SPSite(siteCollectionUrl))
  {
    using (SPWeb rootWeb = siteCollection.RootWeb)
    {
        // Create a new child web in the site collection using the input parameters defining the child web.
        SPWeb newWeb = rootWeb.Webs.Add(webUrl, title, description, 1033, webTemplate, useUniquePermissions, false);
        newWeb.Dispose();
     }
  } 
}

or this:

using (SPSite siteCollection = new SPSite(siteCollectionUrl))
{
  using (SPWeb rootWeb = siteCollection.RootWeb)
  {
    foreach (row in rows) 
    {
        // Create a new child web in the site collection using the input parameters defining the child web.
        SPWeb newWeb = rootWeb.Webs.Add(webUrl, title, description, 1033, webTemplate, useUniquePermissions, false);
        newWeb.Dispose();
     }
  } 
}

My tests were on a VM-hosted single server farm. The VM is running on a laptop, so not too much performance here. Each subsite took an average of 18 seconds to create, and while a bit random (17.5 - 18.25 seconds), showed no up or down trend. There were occasional higher times (25 seconds for example), which is to be expected on a server running background processes and timer jobs. RAM for the Console app grew from 300MB to around 1 GB during the run. (I paused it once, and RAM started going down, so the GC is not keeping up.) I only ran this for 500 loops as you saw a big difference by then.

UPDATE: I was running the console application in debug mode. Without debug mode the time to create a subsite is just over 7 seconds, and is relatively consistent. (No upward trend.)

Here's my code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.SharePoint;

namespace ConsoleApplication7
{
    class Program
    {
        static void Main(string[] args)
        {
            string siteCollectionUrl = "http://sp2016/sites/manysubsites";
            using (SPSite siteCollection = new SPSite(siteCollectionUrl))
            {
                using (SPWeb rootWeb = siteCollection.RootWeb)
                {
                    string webTemplate = rootWeb.WebTemplate;
                    bool useUniquePermissions = false;

                    for (int i = 0; i < 500; i++)
                    {
                        DateTime t1 = DateTime.Now;
                        string title = "sub" + i.ToString().PadLeft(4).Replace(" ", "0");
                        string description = title;
                        string webUrl = title; 

                        // Create a new child web in the site collection using the input parameters defining the child web.
                        SPWeb newWeb = rootWeb.Webs.Add(webUrl, title, description, 1033, webTemplate, useUniquePermissions, false);
                        newWeb.Dispose();
                        DateTime t2 = DateTime.Now;
                        Console.WriteLine("Site: {0}, Time: {1}", title, (t2 - t1));
                    }
                }
            }
        }
    }
}

Other things to consider:

  • Templates and features. Which template are you using? Does it have Features that trigger additional work in the creation of the web?
  • Search. Each site will be crawled by search. Can you pause search while creating these sites and see if that makes a difference? (Search is paused on my test VM.)
  • Custom Tasks or Event Receivers that are responding to new SPWebs.
  • Other activity on the production servers. What % CPU and RAM are being used, both before and during running of your code? (During my tests I saw little change in CPU and RAM over the run of the code, which implies the biggest impact is on the SQL server, not the web servers.)
  • The 5000 limit. Once you reach the 5000 limit, pages like Site Contents may no longer be able to display the list of the subsites. (I'll rerun my code to create more than 5000 to see what else is impacted.)
  • You are using external blog storage... I do not have that in my tests.

So, my "answer" is:

  • If you are not recreating the SPSite or SPWeb (for the rootweb) in each loop, then it's not your code or the basic process.
  • It could be search.
  • It could have something to do with the load balanced servers. (creating locks on each other? Unlikely, but possible.)
  • It could have to do with a Feature in the template.
  • 1
    I changed the template type to "blank site" and let it run to over 5000 subsites. It ran at a consistent 4-4.3 seconds for each site, with an occasional blip due to other processes. – Mike Smith - MCT - MVP Jan 22 '18 at 17:09
  • #mike-smith-mct-mvp Thanks for your detailed answer and follow up comment. I am trying to follow up on the various points you have offered. As we both realize, the environment I am dealing with has a lot more moving parts, which is why I tried to isolate the problem. The C# code snippet using SOM exits (as an application) after each child site creation, so it should not be holding any resource for long given that I am using using for the SPSite and SPWeb objects and then Dispose() on the new SPWeb just then created. – Web User Jan 22 '18 at 17:47
  • It sounds like 18-20 seconds is not excessive for a team site template (New-SPWeb cmdlet is much faster, even with the same template), but the rate decreases as the number of sites in the site collection increases. That is of significant concern, along with the error I am encountering in Event Viewer (which I pasted at the end of my question). – Web User Jan 22 '18 at 17:48
  • When is a blank site template (STS#1) preferred over a team site template (STS#0)? I was using Team Site Template (STS#0) in MOSS 2007 and now that I am migrating to SP 2016, I wonder if the template selection has a broader impact on my provisioning strategy. The goal is to start with a site and customize the lists and libraries, and add custom web parts to a custom web part page, significantly modifying the original site's look and feel, and potentially OOTB features. This worked well in MOSS 2007, but maybe this aspect needs to be reconsidered in SP 2016. – Web User Jan 22 '18 at 17:56
  • 1
    The choice of a template only determines the starting list of lists, libraries, web parts and enabled features. The best practice would be to start with the minimum needed. The Blank Site is no longer available in the UI, but can still be created using code or PowerShell. It does not contain any user lists or libraries. In 2013 and later, I would generally start with a Team Site as it has all of the common lists and libraries like Site Pages and Site Assets, and only one user library, Documents. – Mike Smith - MCT - MVP Jan 22 '18 at 18:17
1

Can you check the usage report on your site collection and if you can disable that during the site creation time. Increase the site Quota until the script finish might help as well. You can stop the search service and search crawl during the script running if possible, for sure you must stop the incremental crawl if it is running. You can increase the speed 10 times more if you go with multithreading, call the site creation method in multithreading, as shown below. But first you can start with two, four, and six and gradually increase the number of thread, be aware about your server memory and resource as it will consume much resource. Please check if haven’t turned the Second Stage Recycle bin off or set a site quota. If a site quota has not been set then the growth of the Second Stage Recycle bin is not limited... The second stage recycle bin is by default limited to 50% size of the site quota, in other words if you have a site quota of 100 GB then you would have a Second Stage recycle bin of 50 GB. If a site quota has not been set, there are not any growth limitations...

Param($Command = $(Read-Host "Enter the script file"), [Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]$ObjectList, $InputParam = $Null, $MaxThreads = 10, $SleepTimer = 200, $MaxResultTime = 120, [HashTable]$AddParam = @{}, [Array]$AddSwitch = @() ) https://gallery.technet.microsoft.com/scriptcenter/Multi-threading-Powershell-d2c0e2e5

  • 1
    I don't think the Site Quota change will impact site creation times. I am curious about the multithreading... have you tested this with SharePoint site creation? The bottleneck appears to be with SQL, not CPU or processes. – Mike Smith - MCT - MVP Jan 22 '18 at 19:15
  • Is there any way to increase logging and trace everything that happens from the C# console application all the way up to the database (including when and how long it takes for the database to perform operations related to the new site creation)? – Web User Jan 22 '18 at 20:30
  • 2
    Not really. The SharePoint logs will show each request, and any errors generated. In a non-production farm you could use the SQL Server Profiler to watch the SQL being processed. Your DBA can monitor all kinds of SQL activity to watch for performance issues. – Mike Smith - MCT - MVP Jan 22 '18 at 21:23

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.