Custom Timer Job not running on SharePoint 2013

Question

All of the custom timer jobs are not running without showing any error/correlation IDs. Below is the only stack I could see from ULS logs even after enabling verbose - no error/failures/exceptions are captured.

03-03-2018 11:42:16.51   w3wp.exe (0x5E18)    0x466C SharePoint Foundation Topology 8xqz    Medium  Updating SPPersistedObject CicoCaseSummaryJob Name=Amlak CICO Report Job. Version: 6456266 Ensure: False, HashCode: 43137058, Id: d7b3b6f4-9c43-4b98-91dd-740b41fec886, Stack: 
at Microsoft.SharePoint.Administration.SPJobDefinition.Update()     
at Microsoft.SharePoint.ApplicationPages.JobEditPage.BtnEnableJob_Click(Object sender, EventArgs e)     
at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)     
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)     
at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)     
at System.Web.UI.Page.ProcessRequest()     
at System.Web.UI.Page.ProcessRequest(HttpContext context)     
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()     
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)     
at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)     
at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)     
at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)     
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)     
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)     
at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)     
at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)     
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)     
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)       c8cf4f9e-bdbe-c043-f022-28a78278cbdc.
03-03-2018 11:42:16.51          w3wp.exe (0x5E18)    0x466C           SharePoint Foundation           Topology            88b9    Verbose           Determining if the current user is a SharePoint Farm Administrator c8cf4f9e-bdbe-c043-f022-28a78278cbdc
03-03-2018 11:42:16.52          NodeRunnerContent1-02bf3c7c-ee2 (0x0BCC)        0x4818            Search            Logging Correlation Data       77a3    Verbose           Ending correlation.     af960167-f03c-41fb-8866-559bb2adb1a1

We have already tried below:

1. Cleared configuration cache.
2. Redeployed the timer job.
3. Restarted services and even the server.

Note: Default timer jobs are running fine without any issue.

Answer 1

Problem description:

In your SharePoint farm you start to experience that the timer jobs are not running normally. Like custom timer jobs or one-time timer jobs do not run once you try running it. Another good example to identify the problem is to run "Merge-SPLogFile" command that creates one-time timer jobs to collect logs from all the servers in the farm.

Cause:

Although the SharePoint Timer service was started in services.msc, the timer service instance object (this is a SharePoint farm object) may be set to “Disabled”.Use below script to get the status of all the timer instances in the farm.

$farm = Get-SPFarm
$FarmTimers = $farm.TimerService.Instance
foreach ($FT in $FarmTimers)
{
write-host "Server: " $FT.Server.Name.ToString();
write-host "Status: " $FT.status;
write-host "Allow Service Jobs: " $FT.AllowServiceJobs;
write-host "Allow Content DB Jobs: " $FT.AllowContentDatabaseJobs;"`n"
}

Solution 1: Run below PowerShell command to re-provision the instance to "Online" state.

$farm = Get-SPFarm
$FarmTimers = $farm.TimerService.Instances
foreach ($FT in $FarmTimers)
{
write-host "Server: " $FT.Server.Name.ToString();
write-host "Status: " $FT.status;
write-host "Allow Service Jobs: " $FT.AllowServiceJobs;
write-host "Allow Content DB Jobs: " $FT.AllowContentDatabaseJobs;"`n"
}
$disabledTimers = $farm.TimerService.Instances | where {$_.Status -ne "Online"}
if ($disabledTimers -ne $null)
{
foreach ($timer in $disabledTimers)
{
Write-Host -ForegroundColor Red "Timer service instance on server " $timer.Server.Name " is NOT Online. Current status:" $timer.Status
Write-Host -ForegroundColor Green "Attempting to set the status of the service instance to online..."
$timer.Provision()
$timer.Start()
write-host -ForegroundColor Red "You MUST now go restart the SharePoint timer service on server " $FT.Server.Name}}
else
{
Write-Host -ForegroundColor Green "All Timer Service Instances in the farm are online. No problems found!"
}

Solution 2 (only for SharePoint 2016 MinRole setups):

Start the Instance via Central Administration -> System Settings -> Manage Services on Server -> Change the view in the upper right corner to "Any" -> Locate "Microsoft SharePoint Foundation Timer" and klick "Fix".

Reference: SharePoint Timer jobs not running(Specially one-time timer jobs)