2

OK, so I really only post here when I'm down to the last of my nerve trying to solve a problem, and this one definitely fits that bill.

Here's the background: I work as a contractor for a large enterprise, and this enterprise has a dedicated patch team that applies all OS level patches. We were patched recently (nothing major), and all of a sudden, one of my WFE servers is 'out of synch' on its patch level, even though no patches were removed, and all servers in the farm had the same patch applied. My farm consists of 5 SP servers (2WFE [load balanced], CA, SSRS and APP) and a standalone dedicated SQL cluster.

So, I've taken the basic steps you normally would to try and fix this. I rn PSConfig, but its in repair mode (showing that we do not need to finish an upgrade on the server). But when I try to go through with the repair, I get the error saying it cant be repaired until the KBs are installed.

Well, the KBs ARE installed, as evidenced by going to Installed Updates and viewing them. In fact, the updates listed in the CA error (health analyzer > product/patch installation or server upgrade required) are installed on both WFE machines, but only one machine is showing the 'out of synch' errors.

I have tried to manually download and install the patches (which are Dec 2013 CU and SP1 for both Server and Foundation - both of which have been installed for over a year), but when I try to run those, just to see if the patches will even take on the server, I get 'an error occurred while running detection" - So I can't even run it and see if the server thinks it needs the patch (again, even though its already installed)! At this point, I am completely stumped. I have had various issues which I beleive are related to this problem, chief among them is a stalling SPTimer service on at least 2 of my servers, which obviously causes issues across the farm (Crawl Jobs not firing, User Prof Synch not working properly).

Has anyone experienced this issue with SP? I have asked many folks in my network of SP professionals, but no one seems to have ever experienced this. Any input would be great. I can provide screenshots, software versions and such if needed.

  • Can you give little more info about which patch is missing and what farm patch level. may be some screen print. – Waqas Sarwar MVP Dec 18 '14 at 16:29
  • RUn RoiScan tool on all the server and compare them, from this you will know exactly what is missing...technet.microsoft.com/en-us/library/hh221405(v=office.14).aspx – Waqas Sarwar MVP Dec 18 '14 at 16:34
  • Unfortunately, I cannot add any third party software (even MS sanctioned) to this secure environment. I tried to get ULSViewer installed and they denied the request!! Still analyzing ULS logs in the old fashioned way... Really painful! – Brian Frick Dec 18 '14 at 16:35
  • Waqas Sawar: It tells me that SP1 and Dec 2013 CU for Server and Foundation are missing, but I know for a fact they were installed (as I installed them myself) over a year ago. Just one day came along, and I started seeing these errors. Let me see if I can get a screen print pushed up to Imgur to provide context. – Brian Frick Dec 18 '14 at 16:36
  • But our farm patch level is just that - SP1 and Dec 2013 CU for both Foundation and Server. – Brian Frick Dec 18 '14 at 16:37
2

Huge thanks to Reddit User reddit.com/u/waterton76 for showing me this answer. I have yet to implement the fix, but I think its the proper way to fix it. Full comments thread at: http://www.reddit.com/r/sharepoint/comments/2pouzy/productpatch_installation_or_server_upgrade/

I come across this every now and then - pretty freaky - and I actually fixed this only yesterday. The issue is due to the Windows installer cache (c:\windows\installer) being partial cleared and/or corrupted, so SharePoint gets confused about what's installed on the server. Most of the posts you see out there that discuss a bad installer cache will tell you that you need to throw out the server and start over. Bahhumbug to that! Without going into the full steps and history to recognize this being the issue (which I can provide if needed), here's how to fix it. http://kurteichler.wordpress.com/2013/09/11/restore-sharepoint-2010-installer-directory-cache-files/ The steps are in that blog post, but to second it: 1) Copy the installer cache from one of your 'good' servers in the farm to a temp directory on the 'bad' server. (c:\temp\installercachebackup for example) 2) Get the OpUtil.vbs script referenced in the post and put it in the installer cache on the bad server. Since the script in the post is encoded weird, I've put the one I use in my OneDrive as a txt file here (short link): http://1drv.ms/1DQqUAW It's over 6000 lines of goodness. Download the txt file from my OneDrive, and save it as oputil.vbs to c:\windows\installer OpUtil.vbs is an internal Microsoft Support tool, so it's really tough to find out in the wild. You're welcome. 3) Open a command prompt local to c:\windows\installer and run the script. For example using the directory referenced above: oputil.vbs /RepairCache /srestorelocation=c:\temp\installercachebackup This will take the files (MSI files for the most part) from the good config cache that you backed up and transform them to be valid for the bad server. The config cache is unique to each machine based on values in the registry, so you can't just copy the cache from one server to another. The script does what you need to line everything back up. 4) In the SharePoint 2010 Management Shell (powershell) run: Get-SPProduct -Local This will update the SP Farm Configuration Database with the patch chain on the bad server. You will now see PSConfig return a valid status when going through the initial screens as well as seeing a good install status on the 'Manage Patch Status' screen in Central Admin (/_admin/PatchStatus.aspx) CYA Disclaimer - Follow these steps at your own risk. I can't guarantee this will work for you, but I'm 99.9% sure this is your issue. I've seen this before and these are the steps I take to fix it. That script file in my OneDrive is the exact one that I have used multiple times, all of them successful. (I'm weird about getting random scripts from the Internet, too) Good luck! Respond how it goes. This routine normally takes about 15 minutes start to finish.

0

does it show which servers dont have the patch? This may be a little extreme (but you are hear as a last straw right?) but maybe disconnect one of those problematic servers from the farm, finish the PSConfig, then add the server back..

Another idea is to uninstall and reinstall the KB. Also, is there a more recent Service Pack or CU that you can install on top of all the others? Make sure you meet all the prereqs for that CU or patch.

  • It does show me which server doesn't have the patch. I am hesitant to remove it from the farm, just because this is live production application, and we heavily rely on both of the load balanced servers. So, if one were to go down, I'd be in a hot mess of a problem. I have thought about doing the uninstall/reinstall, but the fact that the patches are throwing an error when I try to open them (an error occured while running detection), Im wondering if it may be some flag in the registry or something causing issues. The last thing I want to do is remove patches, and then be unable to reinstall – Brian Frick Dec 18 '14 at 16:21
  • is production down because psconfig won't finish? – Mike Dec 18 '14 at 16:33
  • Nope, it is working just fine. I do seem to have requests coming in from the 'failing' WFE as well (according to ULS logs, there are active sessions). – Brian Frick Dec 18 '14 at 16:43
  • Can you look into uninstalling the KBs and reinstalling? Do you have server backups, or VM snapshots just in case? – Mike Dec 18 '14 at 16:56
  • Yeah, I am going to have to reach out to the backup team to see how far back their backups go. I guess worst case scenario, we just get todays backup and restore it if I break it by trying to uninstall/reinstall the KBs. – Brian Frick Dec 18 '14 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.