Good evening everyone. What I'm about to describe is actually occurring. This is not some weird problem that's the result of custom code or a third party SharePoint Solution. It's an out of the box install that is using the best practices described by Microsoft. The problem is so dumbfounding that I've insisted to my client that they let me blow away the virtual servers that they previously created so that I can recreate them, the intent of which is to ensure all aspects of the environment are known.
Prior to doing so I've rebuilt the farm (yes seriously) at least six times. I've taken this to tier three support with Microsoft, and they provided a solution, which only "broke" again shortly after I was confident that everything was fixed. I assure all of you, this is not a typical problem that people regularly encounter; it makes me think that someone is trying to sabotage my efforts to deploy this farm. With all of that said, here is the scenario:
Fresh install from the ground up including virtual servers. There is an "application server" which is necessary based on requirements of the customer. This is the SharePoint server hosting all of the service applications. The only thing installed on the box is SharePoint and its prerequisites. The OS is Windows Server 2008 R2, which is full patched with the latest updates. SharePoint Service Pack 1 and the April Cumulative Update package is installed.
Second server is what would have been considered a Web Front End with SharePoint 2007. Again, nothing other than SharePoint and the prerequisites are installed, fully patched OS, and no service applications are running. The server references the application server for all of its functionality.
Third server is hosting an SQL Server Enterprise Edition instance. NO SharePoint components have been installed thus far; it is simply hosting the databases. Same OS as the other two server, Windows 2008 R2, fully updated. SQL Server is running with SP1 for SQL Server R2 and the Cumulative Update #6 for R2 SP1.
The last server is running SQL Express R2, patched to the same extent as the other database server. This box is hosting ASP.NET Session State and the State Service database.
Every single time I build out this Farm, everything works without issue. Then, twenty-four hours later I come back to perform another task and I can't edit health and usage report list items. I can view them, but each time I try to edit one of them - for instance disable a rule - I receive an error telling me that an exception occurred. There are not any errors in the application or system event logs, no failure audits in the security event log, and even the ULS logs, running in verbose mode, provide very little insight. The only error I see in the ULS logs is as follows:
System.OutOfMemoryException: Server Out Of Memory. There is no memory on the server to run your program. Please contact your administrator with this problem.
at Microsoft.SharePoint.Library.SPRequestInternalClass.GetUserToken(String bstrUrl, String bstrLogin)
at Microsoft.SharePoint.Library.SPRequest.GetUserToken(String bstrUrl, String bstrLogin)
at Microsoft.SharePoint.SPWeb.GetUserToken(String userName)
at Microsoft.SharePoint.Publishing.CacheManager.<.ctor>b__0(SPSite newSite)
at Microsoft.SharePoint.Publishing.CommonUtilities.<>c__DisplayClass1.<RunWithElevatedSite>b__0()
at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass4.<RunWithElevatedPrivileges>b__2()
at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)
at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)
at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)
at Microsoft.SharePoint.Publishing.CommonUtilities.RunWithElevatedSite(SPSite siteNonElev, Boolean allowUnsafeUpdates, ElevatedSiteProcessor callWithElevatedSite)
at Microsoft.SharePoint.Publishing.CacheManager..ctor(SPSite site)
at Microsoft.SharePoint.Publishing.CacheManager.GetManager(SPSite site, Boolean useContextSite, Boolean allowContextSiteOptimization)
at Microsoft.SharePoint.Publishing.WebControls.CombinedBasePermissions..ctor()
at Microsoft.SharePoint.Publishing.WebControls.ConsoleNode.CurrentPermissions()
at Microsoft.SharePoint.Publishing.WebControls.ConsoleNode.CurrentState(Page currentPage, Boolean cacheResult)
at Microsoft.SharePoint.Publishing.WebControls.EditingMenuActions.ConsoleAction.OnPreRender(EventArgs e)
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
Of course, the server is not out of memory. None of the servers in the farm are out of memory. I've even added memory to all of the servers just in case Performance Monitor was not reporting correctly.
Has anyone run into something even remotely similar to this before? An 80 hour project has been going on for the last six months at this point and I don't have any answers. I would really appreciate any insight that the community could provide.
Thanks in advance for the assistance.