Product/Patch Installation or Server Upgrade Required - SharePoint 2010

Question

OK, so I really only post here when I'm down to the last of my nerve trying to solve a problem, and this one definitely fits that bill.

Here's the background: I work as a contractor for a large enterprise, and this enterprise has a dedicated patch team that applies all OS level patches. We were patched recently (nothing major), and all of a sudden, one of my WFE servers is 'out of synch' on its patch level, even though no patches were removed, and all servers in the farm had the same patch applied. My farm consists of 5 SP servers (2WFE [load balanced], CA, SSRS and APP) and a standalone dedicated SQL cluster.

So, I've taken the basic steps you normally would to try and fix this. I rn PSConfig, but its in repair mode (showing that we do not need to finish an upgrade on the server). But when I try to go through with the repair, I get the error saying it cant be repaired until the KBs are installed.

Well, the KBs ARE installed, as evidenced by going to Installed Updates and viewing them. In fact, the updates listed in the CA error (health analyzer > product/patch installation or server upgrade required) are installed on both WFE machines, but only one machine is showing the 'out of synch' errors.

I have tried to manually download and install the patches (which are Dec 2013 CU and SP1 for both Server and Foundation - both of which have been installed for over a year), but when I try to run those, just to see if the patches will even take on the server, I get 'an error occurred while running detection" - So I can't even run it and see if the server thinks it needs the patch (again, even though its already installed)! At this point, I am completely stumped. I have had various issues which I beleive are related to this problem, chief among them is a stalling SPTimer service on at least 2 of my servers, which obviously causes issues across the farm (Crawl Jobs not firing, User Prof Synch not working properly).

Has anyone experienced this issue with SP? I have asked many folks in my network of SP professionals, but no one seems to have ever experienced this. Any input would be great. I can provide screenshots, software versions and such if needed.

Answer 1

Huge thanks to Reddit User reddit.com/u/waterton76 for showing me this answer. I have yet to implement the fix, but I think its the proper way to fix it. Full comments thread at: http://www.reddit.com/r/sharepoint/comments/2pouzy/productpatch_installation_or_server_upgrade/

I come across this every now and then - pretty freaky - and I actually fixed this only yesterday. The issue is due to the Windows installer cache (c:\windows\installer) being partial cleared and/or corrupted, so SharePoint gets confused about what's installed on the server. Most of the posts you see out there that discuss a bad installer cache will tell you that you need to throw out the server and start over. Bahhumbug to that! Without going into the full steps and history to recognize this being the issue (which I can provide if needed), here's how to fix it. http://kurteichler.wordpress.com/2013/09/11/restore-sharepoint-2010-installer-directory-cache-files/ The steps are in that blog post, but to second it: 1) Copy the installer cache from one of your 'good' servers in the farm to a temp directory on the 'bad' server. (c:\temp\installercachebackup for example) 2) Get the OpUtil.vbs script referenced in the post and put it in the installer cache on the bad server. Since the script in the post is encoded weird, I've put the one I use in my OneDrive as a txt file here (short link): http://1drv.ms/1DQqUAW It's over 6000 lines of goodness. Download the txt file from my OneDrive, and save it as oputil.vbs to c:\windows\installer OpUtil.vbs is an internal Microsoft Support tool, so it's really tough to find out in the wild. You're welcome. 3) Open a command prompt local to c:\windows\installer and run the script. For example using the directory referenced above: oputil.vbs /RepairCache /srestorelocation=c:\temp\installercachebackup This will take the files (MSI files for the most part) from the good config cache that you backed up and transform them to be valid for the bad server. The config cache is unique to each machine based on values in the registry, so you can't just copy the cache from one server to another. The script does what you need to line everything back up. 4) In the SharePoint 2010 Management Shell (powershell) run: Get-SPProduct -Local This will update the SP Farm Configuration Database with the patch chain on the bad server. You will now see PSConfig return a valid status when going through the initial screens as well as seeing a good install status on the 'Manage Patch Status' screen in Central Admin (/_admin/PatchStatus.aspx) CYA Disclaimer - Follow these steps at your own risk. I can't guarantee this will work for you, but I'm 99.9% sure this is your issue. I've seen this before and these are the steps I take to fix it. That script file in my OneDrive is the exact one that I have used multiple times, all of them successful. (I'm weird about getting random scripts from the Internet, too) Good luck! Respond how it goes. This routine normally takes about 15 minutes start to finish.

Answer 2

does it show which servers dont have the patch? This may be a little extreme (but you are hear as a last straw right?) but maybe disconnect one of those problematic servers from the farm, finish the PSConfig, then add the server back..

Another idea is to uninstall and reinstall the KB. Also, is there a more recent Service Pack or CU that you can install on top of all the others? Make sure you meet all the prereqs for that CU or patch.