I have a custom webpart in SP 2013 environment over Amazon. This webpart reads a xml
file, loads the contents to the SQL Database
and then reads the contents from Database and create discussions in a new Community site (again, created by webpart code). On my DEV environment which a single server farm with SQL(2012) on a separate box, the code works fine.
But when i run the code on my TEST and PROD environments, the code starts as expected. It creates community site, loads xml
data to SQL tables and starts creating discussions. But, after creating around 20-40 discussions it gives following error,
The TEST environment have 2 load balanced FE, one APP server, one DC, one ADFS server and one DB server. We are using PING as a Trusted identity provider
on TEST and PROD, but not on DEV.
I am executing the page from a browser on one of the FE servers.
I get following corresponding entries on the SharePoint LOG files,
03/05/2015 06:23:53.98 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server Communities aa797 Monitorable DiscussionListCommunityEventHandler.HandleEvent: begin d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.04 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server Communities aa8ac Monitorable DiscussionListCommunityEventHandler.HandleEvent: success d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.04 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server Communities al3jg Monitorable CategoriesListCommunityEventHandler.HandleEvent: begin d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.07 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server Communities al3jj Monitorable CategoriesListCommunityEventHandler.HandleEvent: success d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.07 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server Communities aa8ae Monitorable ReputationCommunityEventHandler.HandleEvent: begin d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.09 w3wp.exe (0x107C) 0x18EC SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (EnsureListItemsData). Execution Time=21.054048 d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.09 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server Communities al3jd Monitorable CommunityNotificationsEventHandler.HandleEvent: begin d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.12 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server Communities al3jf Monitorable CommunityNotificationsEventHandler.HandleEvent: success d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.40 w3wp.exe (0x107C) 0x124C SharePoint Foundation Database ahjqp High [Forced due to logging gap, cached @ 03/05/2015 06:23:53.98, Original Level: Verbose] SQL connection time: 0.05992 for Data Source=SPTSTSQLServer01;Initial Catalog=WSS_Content_Share;Integrated Security=True;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Connect Timeout=15;Application Name=SharePoint[w3wp][2][WSS_Content_Share] d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.40 w3wp.exe (0x107C) 0x124C SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (EnsureListItemsData). Execution Time=425.786224 d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.40 w3wp.exe (0x107C) 0x124C SharePoint Foundation General al3j4 Monitorable SPUtility.CreateNewDiscussionReply: begin d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.42 w3wp.exe (0x107C) 0x124C SharePoint Foundation Database fa45 High System.Threading.ThreadAbortException: Thread was being aborted. at Microsoft.SharePoint.Utilities.SqlSession.get_ConnectionString() at Microsoft.SharePoint.Utilities.SqlSession.OnPreExecuteCommand(SqlCommand command) at Microsoft.SharePoint.Utilities.SqlSession.ExecuteReader(SqlCommand command, CommandBehavior behavior, SqlQueryData monitoringData, Boolean retryForDeadLock) d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.42 w3wp.exe (0x107C) 0x124C SharePoint Foundation Database fa46 High at Microsoft.SharePoint.SPSqlClient.ExecuteQueryInternal(Boolean retryfordeadlock) at Microsoft.SharePoint.SPSqlClient.ExecuteQuery(Boolean retryfordeadlock) at Microsoft.SharePoint.Library.SPRequestInternalClass.GetListItemDataWithCallback2(IListItemSqlClient pSqlClient, String bstrUrl, String bstrListName, String bstrViewName, String bstrViewXml, SAFEARRAYFLAGS fSafeArrayFlags, ISP2DSafeArrayWriter pSACallback, ISPDataCallback pPagingCallback, ISPDataCallback pPagingPrevCallback, ISPDataCallback pFilterLinkCallback, ISPDataCallback pSchemaCallback, ISPDataCallback pRowCountCallback, Boolean& pbMaximalView) at Microsoft.SharePoint.Library.SPRequestInternalClass.GetListItemDataWithCallback2(IListItemSqlClient pSqlClient, String bstrUrl, String bstrListName, String bstrViewNa... d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.42* w3wp.exe (0x107C) 0x124C SharePoint Foundation Database fa46 High ...me, String bstrViewXml, SAFEARRAYFLAGS fSafeArrayFlags, ISP2DSafeArrayWriter pSACallback, ISPDataCallback pPagingCallback, ISPDataCallback pPagingPrevCallback, ISPDataCallback pFilterLinkCallback, ISPDataCallback pSchemaCallback, ISPDataCallback pRowCountCallback, Boolean& pbMaximalView) at Microsoft.SharePoint.Library.SPRequest.GetListItemDataWithCallback2(IListItemSqlClient pSqlClient, String bstrUrl, String bstrListName, String bstrViewName, String bstrViewXml, SAFEARRAYFLAGS fSafeArrayFlags, ISP2DSafeArrayWriter pSACallback, ISPDataCallback pPagingCallback, ISPDataCallback pPagingPrevCallback, ISPDataCallback pFilterLinkCallback, ISPDataCallback pSchemaCallback, ISPDataCallback pRowCountCallback, Boolean& pbMaximalView) at Microsoft.SharePoint.SPListItemCollection.EnsureListIte... d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.42* w3wp.exe (0x107C) 0x124C SharePoint Foundation Database fa46 High ...msData() at Microsoft.SharePoint.SPListItemCollection.GetItemIndexById(Int32 id) at Microsoft.SharePoint.SPListItem.EnsureItemIsValid() at Microsoft.SharePoint.SPListItem.GetValue(SPField fld, Int32 columnNumber, Boolean bRaw, Boolean bThrowException) at Microsoft.SharePoint.SPListItem.GetValue(String strName, Boolean bThrowIfValueMissing, Boolean bThrowIfFieldMissing) at Microsoft.SharePoint.Utilities.SPUtility.CreateNewDiscussionReply(SPListItem parent) at JiveContentMigration.JiveContentMigrationWebpart.JiveContentMigrationWebpartUserControl.associateAllRepliesToDiscussion(SPListItem newSPDiscusion, String jiveDiscussionID, SPList imgLibrary) at JiveContentMigration.JiveContentMigrationWebpart.JiveContentMigrationWebpartUserControl.createDiscussions() at ... d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.43 w3wp.exe (0x107C) 0x124C SharePoint Foundation Database aek90 High SecurityOnOperationCheck = True d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.43 w3wp.exe (0x107C) 0x124C SharePoint Foundation Database btq8 High System.Threading.ThreadAbortException: Thread was being aborted. at Microsoft.SharePoint.Utilities.SqlSession.get_ConnectionString() at Microsoft.SharePoint.Utilities.SqlSession.OnPreExecuteCommand(SqlCommand command) at Microsoft.SharePoint.Utilities.SqlSession.ExecuteReader(SqlCommand command, CommandBehavior behavior, SqlQueryData monitoringData, Boolean retryForDeadLock) at Microsoft.SharePoint.SPSqlClient.ExecuteQueryInternal(Boolean retryfordeadlock) at Microsoft.SharePoint.SPSqlClient.ExecuteQuery(Boolean retryfordeadlock) d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.43 w3wp.exe (0x107C) 0x124C SharePoint Foundation Database ad194 High ExecuteQuery failed with original error 0x80131530 d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.43 w3wp.exe (0x107C) 0x124C SharePoint Foundation General xxpm High Unable to execute query: Error 0x80131530 d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.43 w3wp.exe (0x107C) 0x124C SharePoint Foundation General 8e2s Medium Unknown SPRequest error occurred. More information: 0x80131530 d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.43 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server User Profiles ajk39 Medium UserProfileDBCache_WCFLogging::Begin ProfileDBCacheServiceClient.GetUserData.ExecuteOnChannel d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.43 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server User Profiles ajk35 Medium MossClientBase_WCFLogging::Begin MossClientBase.ExecuteOnChannel d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.43 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server User Profiles ajk36 Medium MossClientBase_WCFLogging:: MossClientBase.ExecuteOnChannel - Executing codeblock on channel d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.43 w3wp.exe (0x107C) 0x18EC SharePoint Foundation Topology e5mc Medium WcfSendRequest: RemoteAddress: 'http://sptst02:32843/19b4b168d5d94364927b17d33af9d7b4/ProfileDBCacheService.svc' Channel: 'Microsoft.Office.Server.UserProfiles.IProfileDBCacheService' Action: 'http://Microsoft.Office.Server.UserProfiles/GetUserData' MessageId: 'urn:uuid:4894721d-2832-49ef-9525-35c3b1b34492' d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.45 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server User Profiles ajk37 Medium MossClientBase_WCFLogging:: MossClientBase.ExecuteOnChannel - Executed codeblock on channel d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.45 w3wp.exe (0x107C) 0x18EC SharePoint Portal Server User Profiles ajk4a Medium UserProfileDBCache_WCFLogging::End ProfileDBCacheServiceClient.GetUserData.ExecuteOnChannel d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.45 w3wp.exe (0x107C) 0x18EC SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (CommunityEventReceiver::HandleEvent). Execution Time=538.769216 d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.45 w3wp.exe (0x107C) 0x18EC SharePoint Foundation Dev Events ajbpn Medium Set EventFiringDisabled to [False]. Stack trace: [ at Microsoft.SharePoint.Portal.CommunityEventReceiver.HandleEvent(SPItemEventProperties properties) at Microsoft.SharePoint.Portal.CommunityEventReceiver.ItemAdded(SPItemEventProperties properties) at Microsoft.SharePoint.SPEventManager.RunItemEventReceiver(SPItemEventReceiver receiver, SPUserCodeInfo userCodeInfo, SPItemEventProperties properties, SPEventContext context, String receiverData) at Microsoft.SharePoint.SPEventManager.RunItemEventReceiverHelper(Object receiver, SPUserCodeInfo userCodeInfo, Object properties, SPEventContext context, String receiverData) at Microsoft.SharePoint.SPEventManager.<>c__DisplayClassa`1.<InvokeEventReceiver>b__7() at Microsoft.SharePoint.SPSecurity.RunAsUser(SPUserToken userToken... d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.45* w3wp.exe (0x107C) 0x18EC SharePoint Foundation Dev Events ajbpn Medium ..., Boolean bResetContext, WaitCallback code, Object param) at Microsoft.SharePoint.SPEventManager.InvokeEventReceiver[ReceiverType](SPUserToken userToken, Guid tranLockerId, RunEventReceiver runEventReceiver, Object receivers, Int32 n, Object properties, SPEventReceiverOperationsType operationsType, Boolean& canceled, String& error) at Microsoft.SharePoint.SPEventManager.InvokeEventReceivers[ReceiverType](SPRemoteEventReceiverCallType remoteCallType, SPUserToken userToken, Guid tranLockerId, RunEventReceiver runEventReceiver, RunRemoteEventReceiver runRemoteEventReceiver, Object receivers, Object properties, Boolean checkCancel, SPEventReceiverOperationsType operationsType) at Microsoft.SharePoint.SPEventManager.InvokeEventReceivers[ReceiverType](SPRemoteEventReceiverCallType re... d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.45* w3wp.exe (0x107C) 0x18EC SharePoint Foundation Dev Events ajbpn Medium ...moteCallType, Byte[] userTokenBytes, Guid tranLockerId, RunEventReceiver runEventReceiver, RunRemoteEventReceiver runRemoteEventReceiver, Object receivers, Object properties, Boolean checkCancel, SPEventReceiverOperationsType operationsType) at Microsoft.SharePoint.SPEventManager.HandleEventCallback[ReceiverType,PropertiesType](Object callbackData) at Microsoft.SharePoint.Utilities.SPThreadPool.WaitCallbackWrapper(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.QueueUserWorkItemCallback.System.Threa... d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.45* w3wp.exe (0x107C) 0x18EC SharePoint Foundation Dev Events ajbpn Medium ...ding.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch() ] d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.45 w3wp.exe (0x107C) 0x18EC SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (Event Receiver (Microsoft.SharePoint.Portal, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c, Microsoft.SharePoint.Portal.CommunityEventReceiver)). Execution Time=541.011328 d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.45 w3wp.exe (0x107C) 0x18EC SharePoint Foundation Performance nask Monitorable An SPRequest object was not disposed before the end of this thread. To avoid wasting system resources, dispose of this object or its parent (such as an SPSite or SPWeb) as soon as you are done using it. This object will now be disposed. Allocation Id: {1025DF23-8B95-4787-A5CC-0C55DCBD0464} To determine where this object was allocated, set Microsoft.SharePoint.Administration.SPWebService.ContentService.CollectSPRequestAllocationCallStacks = true.
03/05/2015 06:23:54.46 w3wp.exe (0x107C) 0x124C SharePoint Foundation General 8nca Medium Application error when access /JMTS/SitePages/Migratecontent.aspx, Error=Request timed out. d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.46 w3wp.exe (0x107C) 0x124C SharePoint Foundation Runtime tkau Unexpected System.Web.HttpException: Request timed out. d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.46 w3wp.exe (0x107C) 0x124C SharePoint Foundation General ajlz0 High Getting Error Message for Exception System.Web.HttpException (0x80004005): Request timed out. d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.51 OWSTIMER.EXE (0x1520) 0x13B0 SharePoint Foundation Monitoring aeh57 Medium Sql Ring buffer status eventsPerSec = ,processingTime=0,totalEventsProcessed=0,eventCount=0,droppedCount=0,memoryUsed=0
03/05/2015 06:23:54.53 w3wp.exe (0x107C) 0x124C SharePoint Foundation Upgrade aiaih High [Forced due to logging gap, Original Level: Verbose] desiredVersion: {0} d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
03/05/2015 06:23:54.54 w3wp.exe (0x107C) 0x124C SharePoint Foundation General aat87 Monitorable d3c7ef9c-3c99-d059-fdcf-f12ec2e06ad1
On the Event Viewer on the FE server, i can see a lot of 6398 error ,
The Execute method of job definition Microsoft.SharePoint.Publishing.Internal.PersistedNavigationTermSetSyncJobDefinition (ID 6103e3e5-5fc2-49a7-b945-e5d39831d2ca) threw an exception. More information is included below.
The HTTP service located at http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc/actas is unavailable. This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later.
Any help will be greatly appriciated.