Perpetual "Upgrade Available" for the Servers on Farm

Question

[This issue has been raised here several times - I've tried all suggested solutions and none have resolved issue (list of resources at bottom under ULS log) - maybe someone has some new insight...]

CUs are installed on 2013 on-prem single WFE as Farm Admin. However...

Get-SPProduct -local # always returns...
    Object reference not set to an instance of an object.

And...

$spprod = Get-SPProduct
$spprod.Servers | select ServerName, Products, InstallStatus # always returns...
-------    ----------------------------------    ----------------
BWSPxxx    {Microsoft SharePoint Server 2013}    UpgradeAvailable

Applying exhaustive research ( inlcuding @BennySkogberg blog @StefenGrobner blog ) :

• Used FeatureAdmin.exe (very cool tool) to scrub all orphaned or faulty features that could prevent upgrade

• Multiple runs of PSConfig.exe with every combination of flags -noinstallcheck -wait -force -cmd secureresources -cmd applicationcontent ( also ran PSConfig with "affinity 1" when an SPUpdatedConcurrencyException was thrown ) Always returns:

  • 0 unsuccessful configuration settings
  • "Successfully upgraded SharePoint Products" [lies!]

• Running stsadmin both ways (-pv No and -pv Yes) to avoid locks Always returns:

  • [0] content database(s) still need upgrade or cannot be upgraded.
  • [0] site collection(s) still need upgrade.

• Between are periodic Reboots, IISRESET, Timer Service restarts, clear-cache ( what are .PERSISTEDFILE files anyway?!? )

After all that UpgradeAvailable persists and the "get-spfarm | select BuildVersion" never changes. Thanks for taking a look.

Per comment here's the trimmed ULS log. Let me know if more detail is needed.

Topology    75bd    High    UpdatedConcurrencyException: The object SPServer Name=BWSPxxx was updated by another user.  
    Determine if these changes will conflict, resolve any differences, and reapply the second change.  
    This error may also indicate a programming error caused by obtaining two copies of the same object in a single thread. 
    Previous update information: User: zzzNET\spdev_farm Process:PSCONFIG (11020) Machine:BWSPxxx Time:April 12, 2018 04:54:48.0000  
    Current update information: User: zzzNET\spdev_farm Process:powershell_ise (8216) Machine:BWSPxxx Time:April 12, 2018 06:49:37.1559
Topology    8xqy    High    ConcurrencyException: Old Version : 12426071 New Version : 12427266 17db3bf0-ad26-4bf6-8354-c8250f11e2ad
Topology    b3ue    High    Fail to update the auto update settings in local SPServer object. Microsoft.SharePoint.Administration.SPUpdatedConcurrencyException: 
    An update conflict has occurred, and you must re-try this action. 
    The object SPServer Name=BWSPxxx was updated by zzzNET\spdev_farm, in the PSCONFIG (11020) process, on machine BWSPxxx.  
    View the tracing log for more information about the conflict.
Topology    99a8    High    MsiOpenProduct {90150000-1015-0409-1000-0000000FF1CE} failed with 1612  17db3bf0-ad26-4bf6-8354-c8250f11e2ad
Topology    99a8    High    MsiOpenProduct {90150000-1015-0409-1000-0000000FF1CE} failed with 1612  17db3bf0-ad26-4bf6-8354-c8250f11e2ad
Topology    99a8    High    MsiOpenProduct {90150000-1015-0409-1000-0000000FF1CE} failed with 1612  17db3bf0-ad26-4bf6-8354-c8250f11e2ad
Topology    999g    Unexpected  Error processing SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\15.0\WSS\InstalledProducts\90150000-110D-0000-1000-0000000FF1CE. 
    Exception: Version string portion was too short or too long.    17db3bf0-ad26-4bf6-8354-c8250f11e2ad
...
Topology    9vyu    Medium  UpdateProductInfoInDatabase, regProductsQuery =     17db3bf0-ad26-4bf6-8354-c8250f11e2ad
PowerShell  6tf2    High    System.NullReferenceException: Object reference not set to an instance of an object.    
PowerShell  91ux    High    Error Category: InvalidData Target Object  Microsoft.SharePoint.PowerShell.SpCmdletGetSPProduct  Details  NULL  RecommendedAction NULL  17db3bf0-ad26-4bf6-8354-c8250f11e2ad

Contributor Blogs:

• Benny Skogberg - An exception of type Microsoft.SharePoint.Administration.SPUpdatedConcurrencyException was thrown
• Stefan Goßner - SharePoint Patching and Get-SPProduct -local
• Stefan Goßner - Common question: why does the version number on the "Servers in Farm" page not change after installing October CU?

Started here:

• SharePoint show incorrect version number after install CU
• Upgrade required even after product configuration wizard was successfull
• Servers listed with “Upgrade Available” over and over
• Trying to run PSConfig for SharePoint 2013 farm, but it is giving errors
• How to remove orphaned features
• how I can know if I have missing CU using “get-spproduct -local” command
• GET-SPProduct -Local throwing error
• An update conflict has occurred, and you must re-try this action. PSConfig error – SharePoint 2010/2013
• Upgrading Cumulative Update throws 'SPUpdatedConcurrencyException'
• SharePoint 2010 /2013: “An exception of type Microsoft.SharePoint.Administration.SPUpdatedConcurrencyException was thrown” while installing an update
• SPUpdatedConcurrencyException Fix when trying to install June 2011 CU for SharePoint Server

Focused on content above, but scanned everything here too...

• Search: results found containing get-spproduct

Answer 1

Upon closer investigation of this exception stated in the ULS log ( posted above ).

    MsiOpenProduct {90150000-1015-0409-1000-0000000FF1CE} failed with 1612

A failure of 1612 means that previously installed patch files have become corrupted or have been removed from the %windir%\installer\ directory.

…also indicated by this ULS exception:

    Exception: Version string portion was too short or too long.

It turns out that Windows stores a deflated copy of the Office-SharePoint patch files in the %windir%\installer\ folder, if any of these files become corrupt or are deleted then future patches will fail if they cannot find their deflated ancestor.

There are 4 possible resolutions:

1. (best) Restore the installer files ( hopefully from back-up or recycle )

2. (may or may not help) Microsoft tool auto-resolves discrepancies between version information in the registry and \installer\ stored patch version numbers.

3. (will help, if you have multiples servers) Restore-InstallerFiles.ps1 by Ahmad Gad.

   • Run this PowerShell script first with the -ScanOnly parameter which will definitively list all missing patches.

   • Run the script with the -SourceMachine parameter and the script will replace any missing patches to the local \installer\ folder from another source machine. The caveat here is that the good source server must be patched identically to the...

      get-spfarm | select BuildVersion
     

     ...of the defective server. If the patch levels are not identical, then crucial files will be missing.

( it was all too late for me, as... ) None of these solutions could replace the long removed \installer\ files. So it's time to stand up a new instance of 2013 and ( although not done yet ) my goal is to install all dependencies and patches up to the version currently showing on the defective server, then point Ahmad's script at the new instance and see if the script will fill in patch gaps to allow a successful upgrade of the defective server.

I'll report back the success of this strategy.