Sharepoint 2013 popularity trend zero count after patch

Question

Recently we have patched our SharePoint 2013 on-premise farm to Configuration database version: 15.0.4763.1000. We found that after patch all site collections' popularity trend report shows zero count after the date of patch. The counts before patch date are correct.

I cannot find any bug report about it. Could you please advise where to check?

UPDATE: I checked the EventStore folder (C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Analytics_{GUID}\EventStore) contains folders with date before patch. So it means the patch make the EventStore stop update. Also I checked the Usage File Creation at D:\LOGS\RequestUsage\. The ImportProgress.ini file is up-to-date according to last modified date.

In the WSS_Logging database I query: Select top 20 userlogin,RowCreatedTime,DocumentPathOriginalCasing from dbo.RequestUsage (nolock) where RequestType='GET' order by RowCreatedTime desc The last valid record RowCreatedTime is dated before patch.

UPDATE2: Checked again the D:\LOGS\RequestUsage\ImportProgress.ini file is NOT up-to-date on one of WFE server. It stopped update since the day of patch. The same file is up-to-date for other WFE and APP servers. Also in the event viewer for the problematic server, it shows

The Execute method of job definition Microsoft.SharePoint.Administration.SPUsageImportJobDefinition (ID <GUID>) threw an exception. More information is included below.

Index was outside the bounds of the array. 

Seems I am close to the problem but I don't know how to solve it.

Answer 1

Finally the popularity trend is resumed. However for the past few days that have zero count, the statistic cannot be recovered (maybe there is some way to recover but not that I aware).

By default, all the SharePoint log are stored in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\. In my environment, the Log directory was moved to D:\LOGS\. In the Log directory there are many subfolders like AnalyticsUsage, ClientServiceActionUsage, ClientServiceRequestUsage, ExportUsage, FeatureUsage... etc. By checking them one by one, the ImportProgress.ini inside was stopped at a datetime before my patch day.

I guess the reason behind is disk full. The patch files copied to D: and the disk became 100% full. It makes the SharePoint log function stop working and SharePoint cannot handle this situation. After the patch complete, disk space was released. However the log cannot be resumed. Every 5 minutes follow log will show up in Event Viewer.

The Execute method of job definition Microsoft.SharePoint.Administration.SPUsageImportJobDefinition (ID ) threw an exception. More information is included below.

Index was outside the bounds of the array.

After search for days, I decide to remove entire Log directory with follow steps:

1. Disable the Timer job
2. Reboot the server (to release the lock on Log directory)
3. Move everything under log directory to somewhere else (retain the root folder \LOGS\ there)
4. Enable and start the Timer job again. Watch all the subfolder and logs resumed.

After the log resumed, I found the D:\LOGS\RequestUsage\ImportProgress.ini file will be updated every 5 minutes. It should be directly related to the job Microsoft SharePoint Foundation Usage Data Import.

I tried to recover the lost data in follow ways:

1. Stop the Timer service
2. Open the D:\LOGS\RequestUsage\ImportProgress.ini and you will find a date time string here. It showed the last update time like 20160201. Changed it to the date of patch and save.
3. Copied back the .usage files from my backup to D:\LOGS\RequestUsage.
4. Start the Timer service. Every 5 minutes, it will consume part of usage files and the ImportProgress.ini will be updated. I left it for a day. When I come back all the .usage files are gone and ImportProgress.ini is back to "today" (the date I do the recovery)

However the statistic show up in popularity trend report are weird. Normally there will be 100 visits but it shows 10000 visits now. I think the .usage files have gone weird in my case.

Here is two reference that help me understand the problem:

Solving SharePoint 2013 Usage Report Problems

Troubleshooting SharePoint 2013 Web Analytics