Application Discovery Load Balancer Service EndpointFailure (topology.svc)

Question

Short Version: How can I change the "ServiceEndpointUri" of Application Discovery and Load Balancer Service Application Proxy? After changing the servers, it shows the computer name of an old server (which doesn't exist anymore) creating multiple events in Windows Event Log.

A failure was reported when trying to invoke a service application: EndpointFailure
Process Name: OWSTIMER
Process ID: 13248
AppDomain Name: DefaultDomain
AppDomain ID: 1
Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:7ef0d151811845ae8907ddfb3f524183#authority=urn:uuid:bdbf864d34e5409    58698ce45b524e982&authority=https://old-appserver:32844/Topology/topology.svc
Active Endpoints: 1
Failed Endpoints:1
Affected Endpoint: http://new-appserver01:32843/7ef0d151811845ae8907ddfb3f524183/ProfileService.svc

Long Version: We expanded our SP Farm from 2 to 4 Servers. First, we added the 4 new Servers to the farm, migrated the service applications (Search etc.), then we removed the 2 old server by uninstalling SharePoint via "Add-/Remove Programs" (Windows Control Panel).

Of course I had to fix the Distributed Cache Service, but now everything works fine except the Application Discovery and Load Balancer Service Application. As you can see above, the event log shows EventId 8313. And, this PowerShell command:

(Get-SPTopologyServiceApplicationProxy).ApplicationProxies | Format-List *

shows a wrong "ServiceEndpointUri", it's the old Application-Server:

ServiceEndpointUri          : https://old-appserver:32844/Topology/topology.svc

Not enough, in CA the Proxy is not listed under Application Discovery and Load Balancer Service Application, it is placed almost at the bottom of the service applications list, standing alone:

And: executing this command shows me only the 4 new SharePoint Servers, the old appserver doesn't show up...

Get-spserviceinstance | Select TypeName, Server

Can anyone help get getting rid of these eventlogs and fixing my Service Application?

Thank you very much!

Answer 1

There is a way, but it is unsupported and requires directly modifying a row in the objects table of the config database for the farm.

First, grab the GUID of the proxy you need to change using

(Get-SPTopologyServiceApplicationProxy).ApplicationProxies | Format-List *

Then open up SQL Server Management Console, connect to the server and instance that hosts the farm's config database, open up a query window, and issue the following query on the objects table in that database:

SELECT        Id, ClassId, ParentId, Name, Status, Version, Properties
FROM          Objects
WHERE         Id = '[YOUR PROXY GUID HERE]'

That should pull back a single row. You are interested in the Properties column of that row. You can fairly quickly expand that and search for your faulty server. Simply replace that with a known good server. It's probably not a bad idea to copy the original text of the properties column for the entry and save that somewhere just on the off chance you need to reference it.

Save the change by hitting Enter, pop over to Central Administration and refresh Manage Service Applications. The proxy should now be lined up properly.

I have spent time with Microsoft on this and there is no "supported" way to do this. I asked them to open it as a bug because the Configuration Wizard should be the tool that checks for this and updates the config database accordingly. Further, there is no Powershell way to do this because the property itself is read-only. Basically, it appears whoever designed this never anticipated the removal of the first server in the farm, which leaves one in the position of either never removing said server, or struggling with this problem. Even more disturbingly, the problem appears to have very real consequences for performance in the farm, especially with PerformancePoint webparts.

Answer 2

In addition to your answer (which solves the problem): After querying the row in your Config DB, you cannot edit the data. But you can update the content with another SQL query:

UPDATE Objects
WHERE Id = '[YOUR PROXY GUID HERE]'
SET Properties = '[YOUR UPDATED COLUMN CONTENT]'