6

Good evening everyone. What I'm about to describe is actually occurring. This is not some weird problem that's the result of custom code or a third party SharePoint Solution. It's an out of the box install that is using the best practices described by Microsoft. The problem is so dumbfounding that I've insisted to my client that they let me blow away the virtual servers that they previously created so that I can recreate them, the intent of which is to ensure all aspects of the environment are known.

Prior to doing so I've rebuilt the farm (yes seriously) at least six times. I've taken this to tier three support with Microsoft, and they provided a solution, which only "broke" again shortly after I was confident that everything was fixed. I assure all of you, this is not a typical problem that people regularly encounter; it makes me think that someone is trying to sabotage my efforts to deploy this farm. With all of that said, here is the scenario:

Fresh install from the ground up including virtual servers. There is an "application server" which is necessary based on requirements of the customer. This is the SharePoint server hosting all of the service applications. The only thing installed on the box is SharePoint and its prerequisites. The OS is Windows Server 2008 R2, which is full patched with the latest updates. SharePoint Service Pack 1 and the April Cumulative Update package is installed.

Second server is what would have been considered a Web Front End with SharePoint 2007. Again, nothing other than SharePoint and the prerequisites are installed, fully patched OS, and no service applications are running. The server references the application server for all of its functionality.

Third server is hosting an SQL Server Enterprise Edition instance. NO SharePoint components have been installed thus far; it is simply hosting the databases. Same OS as the other two server, Windows 2008 R2, fully updated. SQL Server is running with SP1 for SQL Server R2 and the Cumulative Update #6 for R2 SP1.

The last server is running SQL Express R2, patched to the same extent as the other database server. This box is hosting ASP.NET Session State and the State Service database.

Every single time I build out this Farm, everything works without issue. Then, twenty-four hours later I come back to perform another task and I can't edit health and usage report list items. I can view them, but each time I try to edit one of them - for instance disable a rule - I receive an error telling me that an exception occurred. There are not any errors in the application or system event logs, no failure audits in the security event log, and even the ULS logs, running in verbose mode, provide very little insight. The only error I see in the ULS logs is as follows:

System.OutOfMemoryException: Server Out Of Memory.  There is no memory on the server to run your program. Please contact your administrator with this problem.   
 at Microsoft.SharePoint.Library.SPRequestInternalClass.GetUserToken(String bstrUrl, String bstrLogin)    
 at Microsoft.SharePoint.Library.SPRequest.GetUserToken(String bstrUrl, String bstrLogin)    
 at Microsoft.SharePoint.SPWeb.GetUserToken(String userName)    
 at Microsoft.SharePoint.Publishing.CacheManager.<.ctor>b__0(SPSite newSite)    
 at Microsoft.SharePoint.Publishing.CommonUtilities.<>c__DisplayClass1.<RunWithElevatedSite>b__0()    
 at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass4.<RunWithElevatedPrivileges>b__2()    
 at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)    
 at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)    
 at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)    
 at Microsoft.SharePoint.Publishing.CommonUtilities.RunWithElevatedSite(SPSite siteNonElev, Boolean allowUnsafeUpdates, ElevatedSiteProcessor callWithElevatedSite)    
 at Microsoft.SharePoint.Publishing.CacheManager..ctor(SPSite site)    
 at Microsoft.SharePoint.Publishing.CacheManager.GetManager(SPSite site, Boolean useContextSite, Boolean allowContextSiteOptimization)    
 at Microsoft.SharePoint.Publishing.WebControls.CombinedBasePermissions..ctor()    
 at Microsoft.SharePoint.Publishing.WebControls.ConsoleNode.CurrentPermissions()    
 at Microsoft.SharePoint.Publishing.WebControls.ConsoleNode.CurrentState(Page currentPage, Boolean cacheResult)    
 at Microsoft.SharePoint.Publishing.WebControls.EditingMenuActions.ConsoleAction.OnPreRender(EventArgs e)    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Control.PreRenderRecursiveInternal()    
 at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Of course, the server is not out of memory. None of the servers in the farm are out of memory. I've even added memory to all of the servers just in case Performance Monitor was not reporting correctly.

Has anyone run into something even remotely similar to this before? An 80 hour project has been going on for the last six months at this point and I don't have any answers. I would really appreciate any insight that the community could provide.

Thanks in advance for the assistance.

Improve this question
9
  • 1
    How are you determining that the servers are not out of memory? How much are you allocating for each server?
    – user6024
    May 6, 2012 at 22:32
  • The primary database server has 12GB of RAM allocated, SQL Server can use up to 8GB, leaving 8GB for the operating system. Each of the other servers in the farm have a total of 4GB of memory allocated. I know that the servers are not out of memory because I've used data collectors to trace resource usage.
    – Christopher W. Szabo
    May 7, 2012 at 5:23
  • 1
    8GB is the minimum requirement for production use of SharePoint.
    – user6024
    May 7, 2012 at 13:05
  • 2
    So is editing the health and usage reports the only place you see the issue? Could this be a problem with search? Are you able to run crawls? I'm just trying to think of places where you might get a better error message or start to see a pattern that would narrow down what is broken.
    – Kit Menke
    May 7, 2012 at 13:58
  • 1
    This sounds absolutely ridiculous, but we know how reliable SharePoint error messages are: do you have free space available on the Database server drives? When there's no physical space left, SharePoint throws VERY random errors...
    – James Love
    May 7, 2012 at 20:51

4 Answers 4

Reset to default
7

We finally figured out what was going on with this and I figured I would post an answer to help anyone else who might run into something like this. The environment is virtual. The only thing that I didn't do was build the virtual servers from the ground up. Instead, a Systems Administrator took care of that for me.

Unfortunately, the Systems Administrator did not sysprep the servers. Nothing had been generalized, they were all just clones of the same virtual machine. The strange and unexpected behavior would just "show up" after I added more servers to the farm, all of which had the same machine number.

Once I discovered this and demanded access to vCenter I create a new VM, ran sysprep and generalized it, then used that as the clone for all of the other servers. There wasn't a single problem after that.

Lessons Learned:

  1. Don't blame everything on SharePoint.
  2. Do everything yourself.
  3. Bill customers for every hour, even when they're telling you it's your fault.

Hopefully this will help prevent an 80 hour project from turning into an 800 hour project for someone.

Thanks for all of the help everyone!

Improve this answer
2

Make sure that your publishing cache super user and super reader accounts are correctly configured based on your chosen authN provider, see: http://practicalkungfu.net/2012/05/23/sharepoint-2010-server-out-of-memory-in-uls-log/

Improve this answer
1
  • 1
    This fixed my out-of-memory exceptions. No idea why.
    – Rawling
    Sep 13, 2012 at 13:58
1

Just curious but what if you take SQL Express out of the equation? Also another thing you may want to check is if the company is pushing out group polciy that may be overriding settings for the accounts, such as accounts that may need log on as a service permissions.

Improve this answer
1
  • There are no group policies in place that are impacting the servers, I've run group policy modeling on each of them. You're though is a good one though; the policy might be applied after the farm is stood up which would explain why everything works at first and then breaks. Unfortunately, that's not the cause in this particular case. I've also validated the SQL Express is not the cause of the problem by moving the state and session state service application databases to the main database server.
    – Christopher W. Szabo
    May 7, 2012 at 5:25
1

This is probably a long shot, but we saw issues when SharePoint tried to ensure all of our web.config files were identical on all machines. That rule is set to repair the config files automatically and runs daily. We edited this rule so it did not repair automatically, and it solved a specific issue we were having. It is probably worth comparing the web.configs before the farm fails and after to see if there are any differences that could be causing your issues. As @Kit Menke mentioned in his comment, Search would be a likely suspect as well.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.