SharePoint runaway / random app pool recycling every few minutes

Question

Having an issue on a SharePoint 2010 environment that has been happy (on the most part) for several years. Hoping that someone may have seen this issue. The only related issue was a database outage the day before (yes I guess the two are linked - but I need some pointers on where to look)

For the past few days the farm has become slower and less stable, where it reached the point that it couldn't service requests. There is approx 15 apppools (web apps and sharepoint internal ones) which are recycling during business hours (not just the scheduled early hours), approx 3-4 pools per minute.

Digging in ULS, there is indeed logs along the lines of "The app domain is unloading and going to be recycled". The error message was a recompilation limit of 15 was being reached. So, the global web.config setting has been changed (numRecompilesBeforeAppRestart) to a high value. That kind of helped!

Now the pools are recycling at a rate of 1 or 2 every 10 minutes, with a new error - "CONFIG change BuildResult change - HostingEnvironemnt initiated shutdown". I'm at a loss to try and trace what is actually triggering this - any hints to workout what is triggering excessive recompiles? I suspect something at farm level or host level, as all app pools seem to be part of this random recycle behaviour.

So far have tried;

• Host reboot
• Config XML cache clear and restart (this helped to an extent)
• increasing the numRecompilesBeforeAppRestart as above

Our pre-prod and independant 2013 farms don't have this message outside of when it would be expected (the early hours scheduled recycle).

Any ideas?

Update 1

Exact ULS Log:

02/24/2015 18:07:36.28  w3wp.exe (0x15C0)   0x171C  Web Content Management  Publishing  8fjh    High    AppDomainUnloadListener.Stop(False) called. ShutdownReason=MaxRecompilationsReached, this=39392524  
02/24/2015 18:07:36.28  w3wp.exe (0x15C0)   0x171C  Web Content Management  Publishing  8x0e    Medium  AppDomainUnloadListener.Stop() entered lock(this=39392524)  
02/24/2015 18:07:36.28  w3wp.exe (0x15C0)   0x171C  Web Content Management  Publishing  8x0f    Medium  AppDomainUnloadListener.Stop() about to AcquireWriterLock(), this.countdownSeconds=120  
02/24/2015 18:07:36.28  w3wp.exe (0x15C0)   0x171C  Web Content Management  Publishing  8x0i    High    AppDomainUnloadListener.Stop() finally{{}} block with (timedOut == false), ensured that all LRO thread finished 
02/24/2015 18:07:36.64  w3wp.exe (0x15C0)   0x3904  SharePoint Foundation   General avey    High    The application domain /LM/W3SVC/343354202/ROOT-6-130692747510378766 is unloading and going to be recycled. 
02/24/2015 18:07:36.64  w3wp.exe (0x15C0)   0x3904  SharePoint Foundation   General avez    Medium  Shutdown Reason: Recompilation limit of 15 reached  HostingEnvironment initiated shutdown  HostingEnvironment caused shutdown   

This occurs for various app domains, timing as above - max recompilations at default of 15 I get 3+ recycles a minute, change the value, it gets less frequent. A small snippet (note the timestamps);

02/24/2015 18:03:23.73  w3wp.exe (0x15C0)   0x3904  SharePoint Foundation   General avey    High    The application domain /LM/W3SVC/343354202/ROOT-4-130692743253529486 is unloading and going to be recycled. 
02/24/2015 18:03:44.80  w3wp.exe (0x1FFC)   0x1534  SharePoint Foundation   General avey    High    The application domain /LM/W3SVC/1852429627/ROOT-7-130692746161595134 is unloading and going to be recycled.    
02/24/2015 18:04:42.99  w3wp.exe (0x2B88)   0x2854  SharePoint Foundation   General avey    High    The application domain /LM/W3SVC/781802991/ROOT-3-130692743038811758 is unloading and going to be recycled. 
02/24/2015 18:05:46.44  w3wp.exe (0x15C0)   0x3904  SharePoint Foundation   General avey    High    The application domain /LM/W3SVC/343354202/ROOT-5-130692746006573310 is unloading and going to be recycled. 
02/24/2015 18:05:51.41  w3wp.exe (0x1FFC)   0x1534  SharePoint Foundation   General avey    High    The application domain /LM/W3SVC/1852429627/ROOT-8-130692746245044382 is unloading and going to be recycled.    
02/24/2015 18:07:10.31  w3wp.exe (0x39D0)   0x1C84  SharePoint Foundation   General avey    High    The application domain /LM/W3SVC/988379004/ROOT-5-130692748198288110 is unloading and going to be recycled. 
02/24/2015 18:07:18.94  w3wp.exe (0x2B88)   0x2854  SharePoint Foundation   General avey    High    The application domain /LM/W3SVC/781802991/ROOT-4-130692746827782670 is unloading and going to be recycled. 
02/24/2015 18:07:36.64  w3wp.exe (0x15C0)   0x3904  SharePoint Foundation   General avey    High    The application domain /LM/W3SVC/343354202/ROOT-6-130692747510378766 is unloading and going to be recycled. 

I am seeing csc popping up on each request (even pages I have already visited), which led me to monitoring the temp asp.net folder. It seems to be being spammed continuously, which would indicate why each request is slow.

18:27:10.6881047    csc.exe 13148   FASTIO_WRITE    C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\root\4cb8f67e\5fd9588c\App_Web_taxonomyfieldeditor.ascx.5f1d2a77.xx2_0vho.dll   FAST IO DISALLOWED  Offset: 64, Length: 64
18:27:10.6882869    csc.exe 13148   FASTIO_WRITE    C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\root\4cb8f67e\5fd9588c\App_Web_taxonomyfieldeditor.ascx.5f1d2a77.xx2_0vho.dll   FAST IO DISALLOWED  Offset: 128, Length: 248
18:27:10.6884257    csc.exe 13148   FASTIO_WRITE    C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\root\4cb8f67e\5fd9588c\App_Web_taxonomyfieldeditor.ascx.5f1d2a77.xx2_0vho.dll   FAST IO DISALLOWED  Offset: 376, Length: 120
18:27:10.6886062    csc.exe 13148   FASTIO_WRITE    C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\root\4cb8f67e\5fd9588c\App_Web_taxonomyfieldeditor.ascx.5f1d2a77.xx2_0vho.dll   FAST IO DISALLOWED  Offset: 496, Length: 3,600
18:27:10.6890766    csc.exe 13148   FASTIO_WRITE    C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\root\4cb8f67e\5fd9588c\App_Web_taxonomyfieldeditor.ascx.5f1d2a77.xx2_0vho.dll   FAST IO DISALLOWED  Offset: 4,176, Length: 20,244
18:27:10.6900614    csc.exe 13148   FASTIO_WRITE    C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\root\4cb8f67e\5fd9588c\App_Web_taxonomyfieldeditor.ascx.5f1d2a77.xx2_0vho.dll   FAST IO DISALLOWED  Offset: 4,096, Length: 80

Application logs contain various errors for 'Web Controls' like so;

Load control template file /_controltemplates/GbwMobileDefaultTemplates.ascx failed: Thread was being aborted.

Other controls, among others, include; MySiteTopNavigation.ascx, PublishingConsole.ascx, LookupFieldEditor.ascx, WorkflowStatus.ascx failed

All sound like core SharePoint pages.

CPU / RAM usage have been pretty consistent with utilisation before this started going bad.

My finger in the air guess is something going on with the compilation process (i.e failing) which results in it retrying every time. However, tracking that down would probably require some very low level monitoring of .NET.

App pool recycling is set to the random early hours time that SharePoint populates when it provisions the web application (in this case all between 1AM and 2AM).

Answer 1

Appear to have got to the bottom of this. The compiling was going round in circles due to McAfee AV blocking access to C:\Windows\Microsoft.NET\Framework64\v2.0.50727. Digging deeper this inherited environment didn't have the best practice AV exclusions published by Microsoft applied. Once the exclusions were added, the issue went away.

Still intrigued why a system after this long managed without exclusions, either a dodgy DAT or one of the compiled rscx files got corrupted.