We have a production environment that hosts 3 SharePoint Web Applications. We have 2 WFEs and 1 DB server. These are dedicated servers which we are hosted by a hosting provider.
For almost 2 years we have had no issues but just before christmas this year users started reporting performance issues. We have not made any changes to the server or the custom code base for a while so felt it was strange that we were getting these issues out of the blue.
At first the performance issues were intermittent but then became more common. At the moment it is intermittent on 2 of the Web Apps but consistent on another that also has the largest DB, roughly 8GB in size. It has become 40 times slower on loading out of the box List Views. These are simple OTB lists. It is slow in general, I have noticed that when I RDP onto the server and even access Central Admin it is slow at times and rapid at other times.
We have a test environment which is not dedicated and on virtual servers with 1 WFE and 1 DB. I took stsadm backups for all the web applications from live and restored them on Test and then tested the pages. The pages were about 40 times quicker. I looked at SharePoint Developer Dashboard figures and the figures for the live environment (for the same OTB List View page) was quicker in every method and SQL query but the overall time for the live server page load was 40 times slower. Adding all the individual figures didnt add up to the total elapsed time to load the page so I am not sure what the missing time is.
Our hosting provider has looked into this for almost a month now and they say on the hardware side everything looks fine, the RAM usage is fine, the CPU usage is normal and they cant see an issue so they assume that it must be SharePoint and SQL related.
Has anyone experienced such an issue? I have used profiling tools to see if the bottleneck is in the code but I have found none. I personally dont think this is a SharePoint issue or an issue with the application. I would be grateful if you can advise on how to get to the bottom of this? How do I identify what is causing the performance issues? The site is generally slow, all pages seem to take a long time to load from 4-7 seconds where as on the test site the same pages load in 0.1 seconds.
Just want to add that the site is slow regardless of the traffic on it as I have tested it when it is not in use. They are not heavily used sites in terms of traffic.
Update: Just want to add that I did an iisreset on both live WFE's and now the performance is back to what it used to be.