SharePoint Server Search 14 service disabling itself after SP1 install?

Question

A couple of weeks ago I installed SP1 and the June 2011 CU on my SharePoint 2010 environment and since then the service "SharePoint Server Search 14" has been changing its Startup Type from Automatic to Disabled at seemingly random intervals. I can't find anything in the event log that precipitates the change, just entries that report it.

The first time I noticed this I tried to start it and it failed with a logon error. I re-entered the credentials from my crawl account and started it again, this time it worked. Since then all I've needed to do is change the Startup Type back to Automatic and start the service, with one exception. When I reboot the machine the credentials go back to being incorrect and I have to retype the password.

When I go to Central Administration -> System Settings -> Manage services on server the SharePoint Server Search is listed as Stopped. When I try to start it I get an error:

Message from webpage

Starting the Search Service on a server must be done within the context of a Search Service Application. Click OK to go to the Manage Service Applications page where you can create a new Search Service Application or select a Search Service Application and modify its topology.

I click OK and it brings me to the Manage Service Applications page as stated, so I click on Search Service Application to modify the topology of it. I check all of the entries (Admin, Crawl - xxxx, Databases, and Index Partition - 0 - xxxx) to make sure the credentials are correct and all of them are using Windows Authentication which has always worked in the past. I click Apply Topology Changes and get the following error:

Errors were encountered during the configuration of the Search Service Application.

8/7/2012 10:25:18 AM

I click OK and it brings me back to the main page for the Search Service Application where the Crawl status and Background activity entries up top have both been changed to "The Search Server service is currently offline on server FLSPVM because of a logon failure when attempting to start the service." When I check the services for the machine my SharePoint Sever Search 14 service is stopped and the Startup Type has been changed to Manual. I change it back to Automatic and start it but it fails with a logon error. I retype the password and start it again and it works.

When I check the event log for the time specified in the last error I see these two entries:

Lower:

The OSearch14 service was unable to log on as WSDOMAIN\SP_Crawl with the currently configured password due to the following error:
Logon failure: unknown user name or bad password.

To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC).

Upper:

The SharePoint Server Search 14 service failed to start due to the following error:
The service did not start due to a logon failure.

Obviously this is all pointing to a password problem, except the crawl account is set to never expire nor to ever change the password on its own. What the hell is going on with my search service?

I know this is a huge wall of text, so thanks to anyone who's made it this far. I just wanted to be thorough with everything I know and have already tried.

Answer 1

I've had this problem, before. What I did was deleted the certificates from the MMC, and restarted the machine. It was the October 2011 CU.

Also, did you run the PSConfigUI.exe (SharePoint Products and Technologies Configuration Wizard)? It is required after installing June 2011 CU. See here: http://technet.microsoft.com/en-us/sharepoint/ff800847.aspx

Check to see if the search crawl account has full read access. In CA -> Manage Web Applications -> User Policy

Also: link: http://technet.microsoft.com/en-us/library/cc261796.aspx

Answer 2

Q: Did you rebooted the servers and run the upgrade after Cumulative Updates installation? It is mandatory as this could explain behavior you are having. It is mentioned on the CU download page! Also make sure you Managed Account password has not been changed and to be sure, update it FROM SharePoint, very important as managed accounts stored hashed passwords and if changed from the AD only, SharePoint cannot properly work.

Please also check in the Servers and see if Upgrade level is the same everywhere.

Answer 3

Look at this link: http://blog.mastykarz.nl/sharepoint-people-search-lessons-learned-covering-basics/

Do not use "Automatic Password Change" under Managed accounts for SPSearch, it should only be used for App Pools, there's a SYNC issue with the SecureStore and AD when the passwords change, creating a loop in SharePoint, this article explains is very well.