Sharepoint 2007 farm performance goes down?

Question

Our sharepoint 2007 farm is not working properly and we are facing a huge performance hit, some times it throws an unexpected error too.

What all are the points to check? i.e like

1. Depth of the farm
2. Content db size
3. Database though put
4. Web server performance, i/o, processor, memory...
5. Blob settings
6. Disk space
7. Application pool performance

Even, i've stopped the search completely however still the performance issue exists.

What all are the parameters to check with and best practices to follow?

Update Below are my abservations

1. The underlying connection was closed
2. Potentially excessive number of SPRequest objects (13) currently unreleased on thread 67.
3. General Network Error has occured.
4. Network Bandwidth consumed / day is 50GB.
5. Low Virutal memory issue was repoted.
6. Number of hits per day 25Lac (includes 200, 401, 500)
7. Number of hits While the occured 30K / hour (includes 200, 401, 500)
8. Temp DB, SSP DB Disk I/O > 75K.
9. SSP DB Average I/O in ms > 350.
10. Database Performance is bit slow at all time.
11. All the Databases are in the same Tray.
12. Database connectivity issue was reported.
13. Total Content DB size is almost 2TB.
14. It is a single webapplication with 8 different site collection and the depth of the site collections are too long.
15. Full crawl takes 88hrs to complete while incremental takes 32hrs.
16. Application pool memory goes to 7.5GB out of 12GB of the server memory.
17. It was accessible only for the IIS reset
18. The application pool uses almost 85% of the CPU
19. NIC performance is just fine.
20. 23k hits are reported during the peek hours.
21. I suspect a slow SQL response and application pool crashes during the peek hours.

Let me know your thoughts and suggestions.

Update-2 I found that that application pool consumes memory up to 6GB and tries to recycle however due to the huge amount of incoming requests it fails to recycle along with .Net dll. so it sends the 500 signal to the clients.

I suspect that this could be the reason for the performance issue however do not know how to analysis/decide which page/site/sitecollection causes/consumes more memory or its an issue with workflow.

Any idea, how to analysis/find the memory hungry application? or how do I say, it is because of the heavy load?

Update:3

I could see the below list of errors in the ULS log many times.

1. Publishing: Content deployment job failed. Error: 'Microsoft.SharePoint.SPException: Cannot complete this action. Please try again. ---> System.Runtime.InteropServices.COMException (0x80004005): Cannot complete this action.
2. While initializing navigation, found Page placeholder but object was not found at: /GLOBAL/COMPANY/sompage.aspx
3. List item query elapsed time: 5108 milliseconds, Additional data (if available): Query HRESULT: 0 List internal name, flags, and URL: {9EA036BC-BD0D-4D5D-8EDF-4B2439933179}, flags=0x000000022cdc148c,
4. Error: Failure in loading assembly: MyNamespace.SharePoint.myClass, Version=1.0.0.0,
5. Publishing: Content deployment job failed. Error: 'System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
6. Job 'Distribution List Import Job' failed. It will be re-tried again in 60 second(s). Reason: Failed to obtain crawl status. Techinal Details: Microsoft.Office.Server.UserProfiles.UserProfileException: Failed to obtain crawl status. ---> System.Net.WebException: The underlying connection was closed:
7. (#3: Cannot open file "Resources.en-US.resx" for reading.)
8. Exception caught in Search Admin web-service proxy (client). System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.
9. Trying to store a checked out item (/SITES/Teamsite/PAGES/Default.ASPX) in the object cache. This may be because the checked out user is accessing the page, or it could be that the SharePoint system account has the item checked out.
10. # 20 015: "Can RatingResources.de-DE.resx" not open: No such file or folder with that name does not exist.

Any suggestions?

Answer 1

You need to document when/how your performance is taking a hit. You could have a flood of traffic on your WFE's, SQL is pegging out the CPU, Memmory issue, etc. You need to identify where the perormance hit is occuring.

Are the unexpected errors happening when the are getting the performance issues? You need to identify if this is related or a seperate issue.

If the errors are unrelated you need to dig furhter (assuming the error itself is not helping you):

You will need to review all your logs (Windows, SharePoint, SQL). Can you associate any other errors or informational items with your error? finding a correlating event will help you diagnose your issue.

Answer 2

Here's some other suggestions:

• Has any new custom solutions been added/activated recently? Try running a Performance test (via Visual Studio 2008) during your non peak times to see if there is a memory leak? Someone may not be disposing some unmanaged code properly. (I'm putting my money here)
• Have the transaction logs blown out and need truncating?
• Can you isolate a problem attic WFE? Customise your HOST file on your test PC to hit each individual WFE. You could even do this with the VS Performance test.
• Long shot, but ensure your machine.config file (on the WFEs) does not have debug=true(Scott Guthrie post)
• What about analysing a request via Fiddler and see how many 40X's or 50X's come back.
• Disable virus protection temporarily on the servers and see if that improves things.

Answer 3

Don't forget to check your indexing. I've run into instances where at first, everything was great. Then, once data was being uploaded the full & incremental index schedules were overlapping and causing serious performance issues.