I have problem with SP 2010 performance and I need someone to recheck my troubleshooting steps, validate my findings or suggest anything useful.
I have Intranet portal as follows:
- 1 front end server + 1 index server + 1 SQL server
- NLB was used but now it is disabled
- SharePoint 2010 without SP1
- 1 site collection with ~20 subsites (2 levels deep)
- over 30 custom solutions deployed (anything you can imagine but without custom workflows)
- document libraries and lists with unique item permissions and over 10K items are common
- throttling limit for normal users is set higher then normal (but indexes and filters on views are set for standard 5k limit)
Loading times on one application page that serves as display view of custom content type are increasing. It is very complex one with lots of code behind scenes and 3 DataFormWebParts looking at 3 different lists connected via lookup columns (all lists are under 3k items but with unique item level permissions). Load times are increasing day by day, load times are different for site admins and other users, load times are different depending on time of the day. All other pages accross portal have load times in desirable limits. I have local and testing environment with similar setup (stand alone servers) without loading time issues (but item count and permission complexity is not equal).
What my collegues checked:
- Network in general is working perfectly without any issues or any high traffic
- Server CPU utilization and memory usage are low
What I did:
- played a bit with throttling, indexing, changed all on default levels, lower levels, higher... didn’t found any significant change of load time
- did few code revisions to gain around 10% better performance
- pointed to network department since load times are different according time of the day
- watched opposite wall hoping that everything will speed up if I don’t think about problem
What I really need is some help with my latest test.
I used Fiddler on client and did parallel check with Developer Dashboard, ULS logs and IIS logs.
Example: Page size (compressed): 140K (developer dashboard) ~40K (without it)
Request : 1008.94 ms Execution Time 1009.12 ms
ACTUAL PERFORMANCE -------------- ClientDoneRequest: 13:56:14.277 ServerGotRequest: 13:56:14.278 ServerBeginResponse:13:56:18.552 ServerDoneResponse: 13:56:18.554 Overall Elapsed: 0:00:04.283
13:56:15,18 Starting correlation 13:56:16,19 [Last message] 13:56:19,44 Leaving Monitored Scope
So basically Developer dashboard tells me that my custom code runs for 1s. Fiddler gives me info that server response is almost 3s longer. ULS logs are interesting because there is no logged activity for around 3s (what is exactly going on ?). IIS logs only confirms long response times.
I have also noticed that loading times of pages with sizes under 40K are working much, much faster.
In your opinion is this code issue, network issue or server (IIS) issue?