Custom SharePoint 2010 Timer job - Inconsistent execution behavior

Question

The same question is asked here on msdn.

Background: I have a custom SharePoint 2010 Timer job that reads input data from an external database and then populates a bunch of SP lists in a number of site collections. Information about which data should go to which site collection comes from the input data.

Input data is processed in a sequence. So for example, Countries are processed before data for Locations and so on and so forth. Processing of each kind of data is broken into a separate function. As data is copied into a SP list, the timer job also performs a bunch of actions on it which generally involve running a workflow. This timer job has been running fine for many months. It runs only on the CA server and is scheduled to run every 10 minutes and presently does manage to do so. It also logs message to a custom log file on the file system.

Problem: When updates to this timer job are re-deployed (via .wsp), the new binaries do not take effect immediately despite resetting SharePoint 2010 Timer Service and IIS on all app and wfe servers. I also cleared the SharePoint Cache (on all app and wfe servers) at "C:\ProgramData\Application Data\Microsoft\SharePoint\Config\" to no avail.

Inspecting timerjob log files it revealed that when timerjob starts execution every 10 minutes, it does not start at the beginning of Execute() method. Instead the first message that gets logged is from within a method X() that gets called from within Execute(). Timerjob skips all the other method calls and log instructions that exist before the call to X(). In addition, the first log instruction within X() is also ignored and actual logging starts from the second log message.

There is no conditional statement that would determine the execution path for the timerjob. Everything must execute in sequence. It seems like that in the presenece of a lot of data the operations that are performed on each data item do not finish in time before the next execution starts, hence the next execution instance gets queued up somewhere. And the new binaries will not take effect until all queued up instances are executed.

Still it does not explain why execution of certain functions is randomly skipped. This is also true for when i add new behavior to the timerjob and redeploy it.

I would be much appreciative of your help in understanding this behavior as it is most annoying for my customer and quite puzzling for me.

Thanks!!!

Pseudo code of Execute() method

public override void Execute(Guid targetInstanceId)
{
    try
    {
        NDTrace.WriteLine(logPath, "Version for Custom_Timer is {0}", Assembly.GetExecutingAssembly().GetName().Version.ToString(4));
        NDTrace.Enter(logPath, "Custom TIMERJOB Execute()");
        .....
        .....
        .....
        processA(rootweb, sitecollection)
        processB(rootweb, sitecollection)
        processC(rootweb, sitecollection)
        processD(rootweb, sitecollection)
        processX(rootweb, sitecollection)
    }
}

Answer 1

I'll offer a suggestion but, to be honest I don't like these steps because of my fear that they could interrupt other things going on within the server (knock-on effect).

Try these steps and specifically in this order:

1. Stop the SharePoint 2010 Timer. This can be done a number of ways. If you want to do it programmatically just target and end the OWSTIMER.EXE on the server.
2. In the *C:\ProgramData\Microsoft\SharePoint\Config* folder delete all the XML files.
3. Edit the cache.ini file located in the *C:\ProgramData\Microsoft\SharePoint\Config* location where you removed all the XML files. Change the content to a 1. This is a flag that the cache needs to be repopulated.
4. Restart the timer service.

For a more detailed explanation read this article.

Answer 2

For anyone having a similar issue, here is how i managed to solve it.

1. Empty SharePoint Timer Cache (instructions in comment from MAllen22842).

2. When i retracted and removed the .wsp that deployed timer job, the respective .dll didn't get removed from GAC. The version number scheme had changed for my assembly. So redeployment of new .wsp added another .dll to the assembly's folder in GAC. You need to type "c:\windows\assembly\gac_msil" in the "Run" dialog box to open the folder view of GAC and then navigate to the folder of your assembly. Look inside there might be more than one assembly in there. I deleted the whole folder and then did iisreset and restarted sharepoint timer service. And then redeployed the .wsp again and the latest binaries immediately took effect.

Hope it helps!