SharePoint 2010 Performance troubleshooting

Question

I have problem with SP 2010 performance and I need someone to recheck my troubleshooting steps, validate my findings or suggest anything useful.

I have Intranet portal as follows:

• 1 front end server + 1 index server + 1 SQL server
• NLB was used but now it is disabled
• SharePoint 2010 without SP1
• 1 site collection with ~20 subsites (2 levels deep)
• over 30 custom solutions deployed (anything you can imagine but without custom workflows)
• document libraries and lists with unique item permissions and over 10K items are common
• throttling limit for normal users is set higher then normal (but indexes and filters on views are set for standard 5k limit)

Problem:

Loading times on one application page that serves as display view of custom content type are increasing. It is very complex one with lots of code behind scenes and 3 DataFormWebParts looking at 3 different lists connected via lookup columns (all lists are under 3k items but with unique item level permissions). Load times are increasing day by day, load times are different for site admins and other users, load times are different depending on time of the day. All other pages accross portal have load times in desirable limits. I have local and testing environment with similar setup (stand alone servers) without loading time issues (but item count and permission complexity is not equal).

What my collegues checked:

• Network in general is working perfectly without any issues or any high traffic
• Server CPU utilization and memory usage are low

What I did:

• played a bit with throttling, indexing, changed all on default levels, lower levels, higher... didn’t found any significant change of load time
• did few code revisions to gain around 10% better performance
• pointed to network department since load times are different according time of the day
• watched opposite wall hoping that everything will speed up if I don’t think about problem

What I really need is some help with my latest test.

I used Fiddler on client and did parallel check with Developer Dashboard, ULS logs and IIS logs.

Example: Page size (compressed): 140K (developer dashboard) ~40K (without it)

Developer dashboard

Request : 1008.94 ms
Execution Time 1009.12 ms 

Fiddler

ACTUAL PERFORMANCE
--------------
ClientDoneRequest:  13:56:14.277
ServerGotRequest:   13:56:14.278
ServerBeginResponse:13:56:18.552
ServerDoneResponse: 13:56:18.554
Overall Elapsed:    0:00:04.283

ULS Logs

13:56:15,18 Starting correlation
13:56:16,19 [Last message]
13:56:19,44 Leaving Monitored Scope

So basically Developer dashboard tells me that my custom code runs for 1s. Fiddler gives me info that server response is almost 3s longer. ULS logs are interesting because there is no logged activity for around 3s (what is exactly going on ?). IIS logs only confirms long response times.

I have also noticed that loading times of pages with sizes under 40K are working much, much faster.

In your opinion is this code issue, network issue or server (IIS) issue?

Answer 1

Problem solved and it was a very strange one (or maybe not).

There is custom IHttpModule deployed that handles auditing view on list items. Client wanted to log everything so custom module was developed and deployed. Something like this: Sharepoint (WSS/MOSS) Auditing VIEW On List Items

But if you read my question again you will see that there was problem only on single page (serving as Display Form for custom content type). Everything else was working as expected.

At first it looked like there is problem with SPAudit.WriteAuditEvent Method since Audit table in DB is huge. But again I was wrong. After few code changes and some testing problem was narrowed to this part of code:

foreach (SPListItemVersion version in spItem.Versions){...}

There was requirement that some additional info is logged in Audit log for future use. I know there is performance issue when Deleting Multiple Versions of a List Item but this is first info that iteration of SPListItemVersionCollection can produce performance issue of this magnitude.

Why it was limited on one list? It wasn't, but even this list isn't one with most items it is list with most user interactions - many versions are created.

Answer 2

Looking at your reputation I guess you probably know more then me but have you considered or looked at SharePoint Warm Up Timer Job

SharePoint Warm Up Timer Job Tutorial

It could be something to do with Application Pools as well, as I once heard our SharePoint infrastructure manager was talking about it.

and If there was any problem with code, it must had to effect CPU Usage as in case of memory leakage.

It could also be IIS/SharePoint trying to interact with database, check this link out might help :)

Monitoring Latency and SQL Server Round Trips

While googling i find this for testing SharePoint performance you might want to try

troubleshooting-performance-issues-sharepoint-2010

and are you using any back-end server e.g. K2 blackpearl or something ?

Answer 3

I wrote a couple of blog posts on why SharePoint Installations are typically slow. Most of the time its caused by SharePoint developers assuming SharePoint is a very flexible database and you can put any type of information in there. People dont understand that everything will be stored in a single table. Here is a blog you hopefully find useful: Top SharePoint Performance Problems