I need to create about 8000 sites in a site collection, based on Title and path attributes from a database table with 8000 records. I wrote a simple console based application in C# that utilizes the Server Object Model and the following algorithm:
- Connect to this database and retrieve all the rows in this table
- Connect to the desired SharePoint Site Collection (SPSite)
- For each row in rows, Add a website under SPSite.RootWeb.Webs using Add method
This application is running on one of the web servers, not sure if that matters. Actual code for child site creation:
/*----
* Create the child web in the site collection.
*----*/
using (SPSite siteCollection = new SPSite(siteCollectionUrl))
{
using (SPWeb rootWeb = siteCollection.RootWeb)
{
// Create a new child web in the site collection using the input parameters defining the child web.
SPWeb newWeb = rootWeb.Webs.Add(webUrl, title, description, 1033, webTemplate, useUniquePermissions, false);
newWeb.Dispose();
}
}
The setup is a medium SharePoint 2016 farm with two app servers (MinRole: App + Search) and two web servers (MinRole: Web + Distributed Cache).
The application is still running. At first, creation of the child site was taking about 15 seconds. After about 500 sites, that same operation is taking 40 seconds. Now, at about 1500 sites, site creation is taking 70 seconds. If this pattern keeps growing, I am afraid it will take way too long to create all 8000 sites. What may be the reason for this slow down? Is there any way to optimize and increase the speed of site creation, hopefully dramatically? Because I have to provision several more site collections that are similarly large.
Update 1/20/2018: I re-attempted the same site collection provisioning with a PowerShell based script:
Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction Stop
$ErrorActionPreference = "Stop"
$LogFile = "C:\Development\TestLargeSiteCollectionCreation_$(Get-Date -Format "yyyyMMdd").log"
Function WriteToLog
{
#[CmdletBinding()]
Param([string] $logdata)
Write-Host $logdata; Out-File -filepath $LogFile -inputobject $logdata -Append
}
# Create site collection
$SiteCollUrl = "http://<portal-url>/alsc"
$SiteCollTitle = "A Large Site Collection"
$SiteOwner = "CONTOSO\spadmin"
$WebTemplate = Get-SPWebTemplate "STS#0"
$SiteColl = Get-SPSite -Identity $SiteCollUrl -ErrorAction SilentlyContinue
if (!$SiteColl) {
$result = New-SPSite -Name $SiteCollTitle -Url $SiteCollUrl -Template $WebTemplate -OwnerAlias $SiteOwner
WriteToLog "Created site => $result"
}
else {
WriteToLog "Found site => $SiteColl"
}
Get-Content "C:\Development\childSiteList.txt" | ForEach-Object {
$childpath = $_
$c = Start-SPAssignment
$mc = Measure-Command { $newChildWeb = New-SPWeb -Name $childpath -Url "$SiteCollUrl/$childpath" -Template $WebTemplate -UniquePermissions -UseParentTopNav -AssignmentCollection $c -ErrorAction SilentlyContinue }
if ($newChildWeb) {
WriteToLog "Created case site $SiteCollUrl/$childpath in $($mc.Seconds) second(s)"
}
else {
WriteToLog "Unable to create case site $SiteCollUrl/$childpath"
}
Stop-SPAssignment $c
}
I was previously told by a MS SharePoint Support engineer that SharePoint PowerShell cmdlets are preferred to SharePoint Object Model for this type of provisioning, which prompted me to try the above approach. The script ran marginally faster, but I observed that the rate at the beginning was 11 seconds per child site and by the time the script ended, the rate averaged at 40 seconds.
Does a moderately sized farm comprising of 2 servers with MinRole of Web Front End plus Distributed Cache and 2 other servers with MinRole of Application plus Search and finally a 2-server SQL Server cluster somehow have anything to do with this dismal performance? I have a staging environment with a similar set up but comprising of one server each, that is significantly faster and the rate is uniformly set at 10-11 seconds per site.
Second Update 1/21/2018: Corresponding with the creation of each subsite, I see the following error in Event Viewer (TestLargeSiteCollectionCreation is the name of the C# console application I wrote to provision a large number of sub-sites in an existing site collection):
A failure was reported when trying to invoke a service application:
EndpointFailure
Process Name: TestLargeSiteCollectionCreation
Process ID: 6428
AppDomain Name: TestLargeSiteCollectionCreation.exe
AppDomain ID: 1
Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:d623596b16ff4873bac023833775c635#authority=urn:uuid:92c677282f3f411185886510365a07db&authority=https:TestLargeSiteCollectionCreationFTestLargeSiteCollectionCreationF:32844TestLargeSiteCollectionCreationFTopologyTestLargeSiteCollectionCreationFtopology.svc
Active Endpoints: 3
Failed Endpoints:1
Affected Endpoint: http://:32843/d623596b16ff4873bac023833775c635/ProfileService.svc
As I mentioned previously, the production environment has 4 MinRole servers - App01/02 are App+Search, Web01/02 are Web+Distributed-Cache.
Start-Job
. Once PowerShell consumes that much memory the session will need to be closed and a new session started.