SharePoint Hybrid Search Setup - Issues with Kerberos 3-Tier Farm

Question

I'm having issues setting up the first leg of a Hybrid environment between my SP2013 Production On-Premise Farm with my SharePoint Online. It's a 3-Tier system with Kerberos authentication enabled, configured, and working correctly. I have an NTLM single server based test farm for dev On-Premise that I setup Hybrid Searching with my SharePoint Online; works amazingly well! It has been literally hell trying to get my production Kerberos environment working. My question is for a single SharePoint Online farm, can you connect multiple On-Premise farms for Hybrid Mode? If so, has anyone set this up correctly on a 3-Tier Farm using Kerberos? I just want to search outwards right now so no reverse proxy needed yet.

I ask this because of the Domain Name *.mydomain.com and the SPO Application Id of "00000003-0000-0ff1-ce00-000000000000" are the same for SharePoint Online. Then the credentials (self signed cert or 3rd party cert) are registered to the namespace for that WFE; this is the difference.

Get-MsolServicePrincipalCredential -ServicePrincipalName "00000003-0000-0ff1-ce00-000000000000/*.mydomain.com"

Running this command gets you the app id with the many Namespaces. Each namespace contains the necessary Service Principal Credentials, where the farm cert is stored for STS. When you put the SPO Service Principal Id and the company context id together you create the Name Identifier. You use the Name Identifier and the On-Premise site to view/register the On-Premise Trust. I get a Kerberos authentication error, simply put, it's passing anonymous authentication no matter what I do.

$spocontextID = (Get-MsolCompanyInformation).ObjectID
$spoappprincipalID = (Get-MsolServicePrincipal -ServicePrincipalName $spoappid).ObjectID
$sponameidentifier = "$spoappprincipalID@$spocontextID"
Register-SPAppPrincipal -nameIdentifier $sponameidentifier -site $site.rootweb -displayName "SharePoint Online" 
Get-SPAppPrincipal -nameIdentifier $sponameidentifier -site $site.rootweb

I realize it's the remoting in part for PowerShell. I have tried logging in as the Farm Admin, as a Domain Admin, and as myself (also a domain admin) with no luck around the anonymous authentication issue. I've ran both PSSession modes below and get the same Kerberos errors. I'm executing all of these commands from the WFE server as advised. I'm at a loss as to the reason it's not passing the credentials correctly.

Enable-PSRemoting -force

New-PSSession
--- OR ---
Enter-PSSession -ComputerName [SERVERNAME] -Authentication CredSSP –Credential [DOMAIN]\[USERNAME]

Import-Module MSOnline -force –verbose 
Import-Module MSOnlineExtended -force –verbose
Add-PSSnapin Microsoft.SharePoint.Powershell 

Keep in mind that self signed certs in IIS 7 expire after 1 year. This is a hardcoded setting so having to perform this every year sounds a bit daunting, don't you think?

Farm Restore: After messing around with PowerShell with the many commands and potential solutions I found I began to run into errors when trying to access certain services from Central Administration, always a bad thing. I had to restore my entire farm including all system databases to a point in time prior to starting this Hybrid Search project. After that I updated my managed accounts with new passwords, as I had most likely caused them to fall out-of-sync with AD. I then ran the SharePoint Products 2013 Configuration Wizard and the farm came back up. This was a wonderful test of a full farm restore for a 3-Tier system! My point, just make sure you backup all servers in the farm and all related system databases before you try this Hybrid approach. Let me know if you are successful!

PowerShell Solution: I found a solution on TechNet but it doesn't really seem to be a solution, especially for a SharePoint Farm that relies on Kerberos as its authentication mechanism.

Process and Script Reference: Office 365-Configure Hybrid Search with Directory Synchronization –Password Sync

Here is the script I'm using for various commands:

#Variables
#-------------------------------------------------------------------------------------------------------------------
$stscertpfx="c:\cert\stscert.pfx"
$stscertcer="c:\cert\stscert.cer"
$stscertpassword="[PASSWORD]"
$spcn="*.mydomain.com"  
$spsite="http://[SPSite]"
$spoappid="00000003-0000-0ff1-ce00-000000000000"

#-------------------------------------------------------------------------------------------------------------------
#Do Some Conversions With the Certificates to Base64
#-------------------------------------------------------------------------------------------------------------------
$pfxCertificate = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 -ArgumentList $stscertpfx,$stscertpassword
$pfxCertificateBin = $pfxCertificate.GetRawCertData()
$cerCertificate = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
$cerCertificate.Import($stscertcer)
$cerCertificateBin = $cerCertificate.GetRawCertData()
$credValue = [System.Convert]::ToBase64String($cerCertificateBin)

#-------------------------------------------------------------------------------------------------------------------
#Establish Remote Windows PowerShell Connection with Office 365
#-------------------------------------------------------------------------------------------------------------------
Enable-PSRemoting -force

New-PSSession
--- OR ---
Enter-PSSession -ComputerName [SERVERNAME] -Authentication CredSSP –Credential [DOMAIN]\[USERNAME]

Import-Module MSOnline -force –verbose 
Import-Module MSOnlineExtended -force –verbose
Add-PSSnapin Microsoft.SharePoint.Powershell 

#-------------------------------------------------------------------------------------------------------------------
#Log on as a Global Administrator for Office 365 
#-------------------------------------------------------------------------------------------------------------------
Connect-MsolService

#-------------------------------------------------------------------------------------------------------------------
#Review Service Principal Credential Registrations
#-------------------------------------------------------------------------------------------------------------------
Get-MsolServicePrincipalCredential -ServicePrincipalName "00000003-0000-0ff1-ce00-000000000000/*.mydomain.com"

#-------------------------------------------------------------------------------------------------------------------
#Remove Expired or bad Service Principal Credential Registrations (If Needed...)
#-------------------------------------------------------------------------------------------------------------------
Remove-MsolServicePrincipalCredential -KeyIds @("[KEYID]") -ServicePrincipalName "00000003-0000-0ff1-ce00-000000000000/*.mydomain.com"

#-------------------------------------------------------------------------------------------------------------------
#Register the On-Premise STS (THE CERT YOU CREATED!) as Service Principal in Office 365
#-------------------------------------------------------------------------------------------------------------------
New-MsolServicePrincipalCredential -AppPrincipalId $spoappid -Type asymmetric -Usage Verify -Value $credValue 

#-------------------------------------------------------------------------------------------------------------------
#Add new Service Principal Name (If it doesn't already exist)
#-------------------------------------------------------------------------------------------------------------------
$SharePoint = Get-MsolServicePrincipal -AppPrincipalId $spoappid
$spns = $SharePoint.ServicePrincipalNames
$spns.Add("$spoappid/$spcn") #If we are adding a new Service Principal Name *.mydomain.com or *.our-new-domain-name.com
Set-MsolServicePrincipal -AppPrincipalId $spoappid -ServicePrincipalNames $spns 

#-------------------------------------------------------------------------------------------------------------------
#Get Office 365 Information
#-------------------------------------------------------------------------------------------------------------------
$spocontextID = (Get-MsolCompanyInformation).ObjectID
$spoappprincipalID = (Get-MsolServicePrincipal -ServicePrincipalName $spoappid).ObjectID
$sponameidentifier = "$spoappprincipalID@$spocontextID"

#-------------------------------------------------------------------------------------------------------------------
#Finally Establish in the On-Premise Farm a Trust with the ACS
#-------------------------------------------------------------------------------------------------------------------
$site=(Get-Spsite "$spsite").rootweb
Get-SPAppPrincipal -nameIdentifier $sponameidentifier -site $site.rootweb

$appPrincipal = Register-SPAppPrincipal -nameIdentifier $sponameidentifier -site $site.rootweb -displayName "SharePoint Online" 
Set-SPAuthenticationRealm -realm $spocontextID 
New-SPAzureAccessControlServiceApplicationProxy -Name "ACS" -MetadataServiceEndpointUri "https://accounts.accesscontrol.windows.net/metadata/json/1/" -DefaultProxyGroup
New-SPTrustedSecurityTokenIssuer -MetadataEndpoint "https://accounts.accesscontrol.windows.net/metadata/json/1/" -IsTrustBroker -Name "ACS"

Answer 1

I haven't tried Hybrid Search with kerberos, but I've successfully connected my farm to my SharePoint Online environment. It took a while and I had to add all the online-URL:s as trusted sites in internet explorer to make it work properly. Later on I unintentionally destroyed the Cloud Search Service Applications connection by building a Local Search Service Application. Removing the Local SSA I had to run the downloadable Onboard-CloudHybridSearch.ps1 script again. It's more than 400 lines, so I avoid pasting it in here.

My guess is that this script wasn't available three years ago, and you went through great lengths to make your solution work. If you haven't solved this already - it's definately worth a shot. The good part is that you can run the script over and over agin until you get everything right. Error-messages I've seen are easy to follow and you solve errors quickly.

Second, Microsoft talks about duplicate SPN, which may be the case if you've done this with your test environment AND try to access the same tenant from a different farm. To my knowledge, Hybrid Search is a one-on-one connection and several farms unsupported.

During the configuration of a SharePoint hybrid environment, you use the Windows PowerShell cmdlet Set-MsolServicePrincipal to register an SPN, which identifies your SharePoint on-premises farm to SharePoint Online. A duplicate SPN causes connection attempts to fail and may result in a variety of errors. Frequently, the errors say something about authentication, token signing, or Kerberos.

Reference: Troubleshooting hybrid environments