1

Issue: It is taking more than 250ms to respond also sometime it throws DB connectivity Error, throws low memory alerts every now and then.

Environment Details: In our SharePoint 2007 Farm, basically it a publishing framework with 4 WFEs and an application server (just Excel service runs there). A WFE is used as an Index & Query Server. We are facing a performance issue since a month, it is taking more than 250ms to respond also sometime it throws DB connectivity Error. It has only one web-application with 18 Content DBs with 10 Site collection including the root.

Server Details (All the servers are having the same configuration) 4 Cores - 2.27GZ RAM 8GB

Search: Full crawl takes - 90hrs (weekly) Incremental crawl takes - 6hrs (daily) It has been stopped Temporarily.

Performance Analysis(per server) Number of Hits/ Day 1 Lac - 200 hits 1 Lac - 401 hits .5Lac - others Memory Usage 2GB - 2.5GB Peak Memory 5GB - 6.5GB Page Faults 35K - 37K VM size 2GB - 2.2GB I/O Read Bytes 2GB I/O Write Bytes 8GB

SQL Analysis: I could find huge number of "ASYNC_NETWORK_IO" waiting tasks, it should also cause the performance.

EventViewer Analysis: Event ID : 1013 Description : IIS waits for the pool to do a graceful restart that is complete all the pending requests on the pool, shut it down and start again. But if the pending queue is long or the process has hung, the graceful restart will fail and IIS will just do a forced recycle on the pool.

Event ID : 1039 Description : A specific w3wp.exe process caused an error. Either it could be because of the connectivity issue or the process itself crashed.

Event ID : 1039, Event code 3005 Description : Could have been an issue with SharePoint DB

Event ID : 2003 Description : It has taken too long to refresh the W3SVC counters, the stale counters are being used instead.

Event ID : 4830 Description : Low virtual memory.

Event ID : 4830 Description : High memory usage in the W3wp.exe file on a computer that is running Windows Server 2003

Event ID : 9511 Description : An unexpected SQL Server database error occurred while the Windows SharePoint attempted to communicate with the database

Event ID : 10031 Description : Database out of Space, DB Admin Failure

Event ID : 10034 Description : Database server is not accessible

Event ID : 10036 Description : Problem's origin in NIC malfunctioning.

I could see the below list of errors in the ULS log many times.

  1. Publishing: Content deployment job failed. Error: 'Microsoft.SharePoint.SPException: Cannot complete this action. Please try again. ---> System.Runtime.InteropServices.COMException (0x80004005): Cannot complete this action.
  2. While initializing navigation, found Page placeholder but object was not found at: /GLOBAL/COMPANY/sompage.aspx
  3. List item query elapsed time: 5108 milliseconds, Additional data (if available): Query HRESULT: 0 List internal name, flags, and URL: {9EA036BC-BD0D-4D5D-8EDF-4B2439933179}, flags=0x000000022cdc148c,
  4. Error: Failure in loading assembly: MyNamespace.SharePoint.myClass, Version=1.0.0.0,
  5. Publishing: Content deployment job failed. Error: 'System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
  6. Job 'Distribution List Import Job' failed. It will be re-tried again in 60 second(s). Reason: Failed to obtain crawl status. Techinal Details: Microsoft.Office.Server.UserProfiles.UserProfileException: Failed to obtain crawl status. ---> System.Net.WebException: The underlying connection was closed:
  7. (#3: Cannot open file "Resources.en-US.resx" for reading.)
  8. Exception caught in Search Admin web-service proxy (client). System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.
  9. Trying to store a checked out item (/SITES/Teamsite/PAGES/Default.ASPX) in the object cache. This may be because the checked out user is accessing the page, or it could be that the SharePoint system account has the item checked out.
  10. # 20 015: "Can RatingResources.de-DE.resx" not open: No such file or folder with that name does not exist.

There is a workflow which triggers whenever a new post added into the publishing frame work. Farm goes down once the workflow starts.

Any suggestion would be appreciated...!

3 Answers 3

2

There is awfully large amount of information there, some of which is probably not related to the issue with performance.

Are the performance issues demonstrably noticeable on any one page?

Are the memory leak errors traceable to a point in time when a particular page is being used?

Do you have any custom/3rd party code (web parts/workflows/event receivers etc) deployed?

Have you ensured that you don't have lots of 'closed' web parts on any pages which report slow performance?

Its possible that you have some custom code deployed somwehere which has a memory leak in it, this is a common problem with the SharePoint object model, due to the very specific way in which certain objects have to be managed.

There are other possible causes, but I would be focussing on the memory leak first.

3
  • I've taken the memory dump using DebugDiag however seems to be no memory leak issue. I could not see any major issues apart from few "This thread is not fully resolved and may or may not be a problem. Further analysis of these threads may be required." message for the custom DLLs. Even, I've gone through the ULS logs and there is nothing wrong the SPSite/SPWeb objects. Still checking on the IIS Log to find whether it is due to a page or somewhere else..will keep you posted on that. My main concern is that CPU ticks always more than 80% and not sure how to go inside the w3wp process.. Apr 11, 2011 at 10:21
  • I could also see the below message (few times) Thread 25 - System ID 5240 Entry point 0x0547c1c0 Create time 11.04.2011 01:05:30 Time spent in user mode 0 Days 00:22:19.796 Time spent in kernel mode 0 Days 00:02:44.468 This thread is not fully resolved and may or may not be a problem. Further analysis of these threads may be required. Function Microsoft_SharePoint_Publishing!Microsoft.SharePoint.Publishing.BlobCache.RewriteUrl(System.Object, System.EventArgs)+a6e Apr 11, 2011 at 11:55
  • Update: I have done the CPU analysis and found the below: The following threads in 9256_110411_162412.dmp are calling an ISAPI Extension OWSSVR ( 45 47 59 ) 3,70% of threads blocked Apr 11, 2011 at 14:38
2

This is not behavior of a normal SharePoint farm so my first step would be to review any custom code that is included in the Master Page or in commonly used page layouts. If you have any code that runs withinin the http stack, such as an httpModule or a UrlRewriter, I would check that as well.

My next step would be to look at all network components between the server and the database, including NICs, switches and routers. I'm not a network guy, but I'm told that WireShark can shed a LOT of light on a problematic network condition.

After that would be to review the health of the database itself, specifically the NIC and available free space on your DATA drive.

Lastly, if either SharePoint or the Database uses a SAN for storage, it might be worth looking into the health of that as well.

1
  • Below are my latest findigs: Number of Hits during the peak hour - 210625 Successful Hits during the peak hour - 120000 Unauthenticated Hits during the peak hour - 100000 Server has sent - 2.44858GB of data to clients during the peak hour Server has received - 0.82138GB of data from clients during the peak hour. Per day transaction - 23.2919GB (both sent & received) There are hundreds of pages takes more than 10sec to load during the peak hour. I do not know how to conclude that whether the number of hits or the Disk/Network I/O causes the issue... Apr 13, 2011 at 13:05
1

Have you looked into the PAL tool on Codeplex? Run it against all your servers at the same time you're having issues and it should pinpoint where your bottleneck is occurring. It won't tell you if certain pages are the issue but it will tell you which server and which resource is maxed out.

Also, how much content do you have? 6 hours for an incremental and 90 for a full scan seems very long.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.