Can't remove search service

Question

I am unable to remove a search service application from SharePoint 2013 standard edition farm. I do not fully know what caused it to get into this state, but where it is at is this:

• Search is "Paused: for external" - this was done while applying a CU
• There is no active topology - I am thinking this occurred when attempts were made to remove the search service and they failed

This causes the situation that I can't unpause the search because there is not active topology to go to and I can't activate a topology because it is paused.

At this point, I just want to remove the search service completely and rebuild it, but it times out on delete. I have used the UI, I have used Power shell, etc. I have done IIS resets, config cache rebuilds, IIS resets.

I have also not been able to reproduce the exact same situation on my test environment at all.

Answer 1

I ended up contacting MS support of to assist with resolution. The root cause was the timer service "instance" was offline on the server. Since there was only one server in farm, timer jobs no longer ran. The timer service (windows service) itself was running, but the service instance (Farm level on server) was not.

This was found by running the following commands:

$farm = Get-SPFarm
$disabledTimers = $farm.TimerService.Instances | where {$_.Status -ne "Online"}

After that was ran, then we cold use the following commands to change it back to "online"

if ($disabledTimers -ne $null)
{
foreach ($timer in $disabledTimers)
{
Write-Host "Timer service instance on server " $timer.Server.Name " is not Online. Current status:" $timer.Status
Write-Host "Attempting to set the status of the service instance to online"
$timer.Status = [Microsoft.SharePoint.Administration.SPObjectStatus]::Online
$timer.Update()
}
}
else
{
Write-Host "All Timer Service Instances in the farm are online! No problems found"
}

This then allowed the timer jobs to then run. Of course by this time, the One-Once jobs were backlogged and since they did not complete, they did not clean themselves up.

Ran:

Get-SPTimerJob | ?{$_.schedule.description -eq "One-time"} |select displayname,server,locktype,lastruntime | fl

I then just ran to clean up:

$job = Get-SPTimerJob | ?{$_.schedule.description -eq "One-time"}
$job[0].Delete()

After that was completed, I was able to remove he search service and get everything back to running normal.

Note: (I am not sure if it was part of cause, or a result of troubleshooting but the exact same error was replicated in my test environment when I validated these steps) We found that the Job definitions page gave an "object reference not set to an instance of an object" error after that point. I found this article to finish the clean up: http://sharepointviews.com/review-job-definitions-page-error/

This basically consisted of the following commands:

Get-SPTimerJob | ? { $_.DisplayName -eq $null }| select name, Id
$FaultyJob = Get-SPTimerJob <Guid of the job>
$Faultyjob.Delete()

I hope that at least some of this can help someone else that may be struggling.

Answer 2

I have been in this situation before and my suspect is related with your Timer job service. It is recommended you need to suspend Search Service application using below before you install CU update otherwise your search might get corrupted.

$ssa = Get-SPEnterpriseSearchServiceApplication -Identity Search Service Application 
Suspend-SPEnterpriseSearchServiceApplication -Identity $ssa

The following steps hopefully resolve your issue

You might want to try clear configuration cache

use the powershell script in link below: https://gallery.technet.microsoft.com/office/Clear-Configuration-Cache-d55bcba9

Remove search again using powershell.

 $P = Get-SPServiceApplication -identity [GUID of Search Service Application]
 Remove-SPServiceApplication $P -RemoveData

STSADM command if Powershell above not working

 stsadm -o deleteconfigurationobject -id [replace with GUID of enterprise search]

Note: Deleting using stsadm command above, you might need to remove SearchDB and Search Application Pool manually.

It also worth to check if there is an OrphanDB using the following code

$OrphanDBs = Get-SPDatabase | Where {$_.Exists -eq $false}
#clean any Orphans DB
Get-SPDatabase | Where{$_.Exists -eq $false} | ForEach {$_.Delete()}

Once Search service deleted, i have come another problem where i always get timeout when creating Search service either using powershell or Central admin GUI. Then i decide to remove the server from the farm and then rejoin the farm which reinstall the services.