Issue: It is taking more than 250ms to respond also sometime it throws DB connectivity Error, throws low memory alerts every now and then.
Environment Details: In our SharePoint 2007 Farm, basically it a publishing framework with 4 WFEs and an application server (just Excel service runs there). A WFE is used as an Index & Query Server. We are facing a performance issue since a month, it is taking more than 250ms to respond also sometime it throws DB connectivity Error. It has only one web-application with 18 Content DBs with 10 Site collection including the root.
Server Details (All the servers are having the same configuration) 4 Cores - 2.27GZ RAM 8GB
Search: Full crawl takes - 90hrs (weekly) Incremental crawl takes - 6hrs (daily) It has been stopped Temporarily.
Performance Analysis(per server) Number of Hits/ Day 1 Lac - 200 hits 1 Lac - 401 hits .5Lac - others Memory Usage 2GB - 2.5GB Peak Memory 5GB - 6.5GB Page Faults 35K - 37K VM size 2GB - 2.2GB I/O Read Bytes 2GB I/O Write Bytes 8GB
SQL Analysis: I could find huge number of "ASYNC_NETWORK_IO" waiting tasks, it should also cause the performance.
EventViewer Analysis: Event ID : 1013 Description : IIS waits for the pool to do a graceful restart that is complete all the pending requests on the pool, shut it down and start again. But if the pending queue is long or the process has hung, the graceful restart will fail and IIS will just do a forced recycle on the pool.
Event ID : 1039 Description : A specific w3wp.exe process caused an error. Either it could be because of the connectivity issue or the process itself crashed.
Event ID : 1039, Event code 3005 Description : Could have been an issue with SharePoint DB
Event ID : 2003 Description : It has taken too long to refresh the W3SVC counters, the stale counters are being used instead.
Event ID : 4830 Description : Low virtual memory.
Event ID : 4830 Description : High memory usage in the W3wp.exe file on a computer that is running Windows Server 2003
Event ID : 9511 Description : An unexpected SQL Server database error occurred while the Windows SharePoint attempted to communicate with the database
Event ID : 10031 Description : Database out of Space, DB Admin Failure
Event ID : 10034 Description : Database server is not accessible
Event ID : 10036 Description : Problem's origin in NIC malfunctioning.
I could see the below list of errors in the ULS log many times.
- Publishing: Content deployment job failed. Error: 'Microsoft.SharePoint.SPException: Cannot complete this action. Please try again. ---> System.Runtime.InteropServices.COMException (0x80004005): Cannot complete this action.
- While initializing navigation, found Page placeholder but object was not found at: /GLOBAL/COMPANY/sompage.aspx
- List item query elapsed time: 5108 milliseconds, Additional data (if available): Query HRESULT: 0 List internal name, flags, and URL: {9EA036BC-BD0D-4D5D-8EDF-4B2439933179}, flags=0x000000022cdc148c,
- Error: Failure in loading assembly: MyNamespace.SharePoint.myClass, Version=1.0.0.0,
- Publishing: Content deployment job failed. Error: 'System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
- Job 'Distribution List Import Job' failed. It will be re-tried again in 60 second(s). Reason: Failed to obtain crawl status. Techinal Details: Microsoft.Office.Server.UserProfiles.UserProfileException: Failed to obtain crawl status. ---> System.Net.WebException: The underlying connection was closed:
- (#3: Cannot open file "Resources.en-US.resx" for reading.)
- Exception caught in Search Admin web-service proxy (client). System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.
- Trying to store a checked out item (/SITES/Teamsite/PAGES/Default.ASPX) in the object cache. This may be because the checked out user is accessing the page, or it could be that the SharePoint system account has the item checked out.
- # 20 015: "Can RatingResources.de-DE.resx" not open: No such file or folder with that name does not exist.
There is a workflow which triggers whenever a new post added into the publishing frame work. Farm goes down once the workflow starts.
Any suggestion would be appreciated...!