Yesterday, we suffered with the Global outage on our SharePoint 2010 farm as the "User Profile Service Application" was inaccessible during those hours.
Exception:
01/13/2012 08:41:20.16 OWSTIMER.EXE (0x2E44) 0x11DC SharePoint Portal Server User Profiles hyc6 High Exception occured while connecting to WCF endpoint: System.TimeoutException: The request channel timed out while waiting for a reply after 00:00:19.9843750. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://mySAFserver:32843/7512455d976346808bf682dc83abe0f3/ProfilePropertyService.svc' has exceeded the allotted timeout of 00:00:20. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
Solved:
Restarted the "User Profile Sync service" which solves the issue.
Wonder, If I could find the reason why the "User Profile Application" was inaccessible.
suspecting the below reasons:
- System might be suffered with the low resources (Ram/CPU/IO)
Logs: 01/13/2012 09:00:01.68 OWSTIMER.EXE (0x2E44) 0x2610 SharePoint Foundation Health 2137 Critical The SharePoint Health Analyzer detected an error. Drives are running out of free space. Available drive space is less than twice the value of physical memory. This is dangerous because it does not provide enough room for a full memory dump with continued operation. It also could cause problems with the Virtual Memory swap file: SYSTEM (mySAFserver - C:).
however, the system has enough(15GB of freespace) space in "C:\" drive and 5GB of available RAM, so we may ignore this point.
- Application pool could have been stopped/crashed.
Logs:Either the User Profile Application (UPA) service is offline and cannot service this request, or the current application pool identity does not have rights to connect. In certain configurations, you may need to publish the UPA from the Manage Service Applications page in Central Administration. \n The UPA says: "System.TimeoutException"
Logs:
UserProfileServiceUserStatisticsWebPart:LoadControl failed, Exception: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: This User Profile Application's connection is currently not available. The Application Pool or User Profile Service may not have been started. Please contact your administrator. at Microsoft.SharePoint.Portal.UserProfiles.AdminUI.ProfileAdminPage.get_CurrentApplicationProxy() at Microsoft.SharePoint.Portal.WebControls.UserProfileServiceStatisticsWebPartBase.LoadControl(Object sender, EventArgs e)
however, there was no clear indication against the application pool, so we may ignore this point as well.
- Database coud have not accepted the new connection request due to the huge amount of existing connections.
Logs: ConnectionString: 'Data Source=MUVMP168;Initial Catalog="UPA Profile DB";Integrated Security=True;Enlist=False;Asynchronous Processing=False;Connect Timeout=15' ConnectionState: Closed ConnectionTimeout: 15
We have observed huge amount of SQL session was opened(3K) during that time however could not find enough details on the SQL front. Also, there is no direct log states an issue with the database connectivity.
- Network could have been down/unresponsible.
No clue found on this.
