3

We have a 2 server SharePoint 2010 installation, one FE one Index. Each is sitting on a different VM host. Each host has 256 gigs of ram with 4 8 core processors (32 cores). Each box is not over subscribed for either resource.

The 2 VMs have 16 gigs of ram each, and 4 virtual cores. SQL server is running clustered on separate VMs (also on separate hosts from each other and the SP VMs). The DB is on a very fast high capacity and high i-ops SAN with a 10 gig backbone. Performance from the SQL server peaks out at just under 800MBps (bytes not bits) and only comes close to that limit during the night when backups, consolidations and other tasks are being performed. The SAN sits at about 20% of peak i-ops.

The 2 SP boxes average 7 gigs of ram of usage, and 40% CPU utilization; but we are seeing page reloads in the order of 3 to 8 seconds, no matter what we are doing. Clicking on a list settings page, refreshing the front page; it doesn't matter.

Total content is around 2TB. Any thoughts on what might be happening? It's confounding us. Whats worse is this is not in production yet, with only a few users (IT) on it.

Edit:
@Alex I did not see that one. Checking it out now.

@James I'll take a look and report back. The clients are running i5s with 4 gigs of ram and they are on our local network (100mb).

We are using Hyper-V rather than ESX

Edit #2:
Here is the developer dashboard. I'm starting to lean on the Claims Authentication issue. The other 2 alerts only occur on this page, but the Claims issue occurs on every page, on every site.

Additionally, in the upper right and on our mysites our names do not resolve, they have \. In look up fields, if we click the resolve button, they do resolve.

: developer dashboard

6
  • Did you see that question here on stackexchange? Might help. sharepoint.stackexchange.com/questions/7293/…
    – AlexPoint
    Jul 19, 2011 at 15:33
  • Any clues in the Developer Dashboard?
    – James Love
    Jul 19, 2011 at 15:35
  • Also, how is the client performance? SP2010 is Javascript Intensive, and with an older browser it can seem to take a while to load pages.
    – James Love
    Jul 19, 2011 at 15:36
  • You say you have 2TB of content; where is this data located? Is it all in one content database? Also do you have any customisations on said pages? Jul 19, 2011 at 20:21
  • @Benjamin No, it's spread across multiple databases. Most DB sizes are under 110 gigs. There are some customizations, but not on all pages, and we are experiencing the slowness on every page. Jul 19, 2011 at 21:51

3 Answers 3

2

You need to isolate the impact a bit more. Do you have the same issue when loading pages on the server itself?

If it is on the server, than as you do as @James recommends and check out the developer dashboard on the offending pages (you'll should do this for the client side too). If it is every page, you need to start collecting information from performance counters (on the VM themselves). http://technet.microsoft.com/en-us/library/ff758658.aspx

If the server renders fine, now you have moved the issue out between your VM virtual NIC and the Client.

I would load up Fiddler on a client and look at the request times. You should be able to see where the delay is, server side or client side.

Because you may have a network issue (Stressed Router, Slow links, Firewall, etc) you can utilize NetMon and/or WireShark on the client, server or both (best results for analysis).

VMware has a performance troubleshooting paper for ESX as well. http://communities.vmware.com/servlet/JiveServlet/download/10352-2-28235/vsphere4-performance-troubleshooting.pdf

Some really good info here on TS SharePoint Issues.:

http://blogs.technet.com/b/patrick_heyde/archive/2010/05/08/sharepoint-performance-troubleshooting-part-2.aspx

http://blogs.msdn.com/b/russmax/archive/2009/01/28/test.aspx

Edit

I would continue to monitor your performance counters. I have not had a chance to use SPDiag 3.0, but it has some pre-built reports around some of the those counters and the usage database. http://technet.microsoft.com/en-us/library/hh144782.aspx

(I'm assuming yours still using the NTLM claims provider here) You may want to check that your DNS is pointed to a local DNS server (if possible)

Additionally (if you have a large AD deployment) you can verify the DC your servers are using to authenticate by opening a command prompt and typing set lo. I have not experienced it with a server, but have had client machines try to authenticate to a separate DC in another site causing authentication delays / retries. If you are not using a local DC you can do some more extensive troubleshooting at this level. http://support.microsoft.com/kb/247811

Another thought is to look at your machine anti-virus. How active is it. You may want to be sure you have excluded files per Microsoft here: http://support.microsoft.com/kb/952167

You can try and ease the pain why you work it out by ensuring dynamic compression is enabled in IIS, and you utilize the output cache if you have publishing turned on, and ensuring your file system is defragmented.

2
  • I'll check those links. We are using HyperV rather than ESX. Jul 19, 2011 at 18:00
  • Also, yes; same issues on the server itself. The central administration pages load extremely fast though. Jul 20, 2011 at 13:28
1

40% CPU is awfully high for just a handful of users on a VM that powerful. You should look into your Search settings for your farm and make sure it isn't trying to do a continuous full crawl of your 2TB of data. A farm with one WFE being indexed will perform precisely as you describe.

Your host machine has the resources available so why not just create an extra WFE or 2 and load balance them? Not only would this help performance but it would also speed up indexing by spreading the load out.

2
  • 1
    You might want to consider adding a dedicated server for crawling if search does turn out to be your issue. Jul 19, 2011 at 20:23
  • Adding another WFE would incur large costs for add-on licensing e.g. Nintex, Bamboo etc. We do have a second server which does the crawling + search. Sep 23, 2011 at 15:19
0

Without much to go on, this sounds as though it could be a problem with authentication. Check where your domain controller is (especially if using NTLM) and make sure it isn't overloaded.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.