I have a SharePoint 2013 on premise farm that recently started experiencing high CPU usage on the WFE. The server has been locked at 99% for days on end now and the services that are causing it are all workflow related:
We are not sure what is causing this. We cannot seem to find any workflows that are running off the rails which is the only thing we can think of that would cause this. The workflow farm appears to be in a healthy state and reconfiguring it isn't really an option at this time as that would terminate all running instances of workflow that our company heavily relies on. Our farm consists of 1 WFE, 1 App server, and 1 Database server. Workflow is configured on the WFE and yes we are aware that this is not best practice.
Here are two events that started getting thrown as soon as the high CPU usage began:
ServiceBus threw MessageNotFound exception while trying to receive a deferred message. Deferred message context details: MessageId = 573074, ActionNamespace = Microsoft.Workflow.Management, ActionName = Terminate, InstanceId = {6b339f70-34ce-464a-8b62-e27dea949015}, ResumptionKey = (null), MessageTraceId = {44259071-166f-4d87-b770-d26787e35b85}
/sharepoint/default/81f1359d-754d-4ef9-b013-c63410d5762b/156d509e-da00-4f68-be8e-2ded413ffb90/53a8f467-11ff-ff92-ee48-0070fd375256_dbbeb1cc-941b-4cfb-a778-59bd88be507d: The instance attempted to process the same message too many times. The message may be causing an unexpected failure. The limit is controlled by the MaxMessageProcessingAttemptsOnUnknownFailure and MaxMessageProcessingAttemptsOnKnownFailure system configuration values (currently set to 10 and 1000). The actual values seen are 0 and 1000.
Any insight on this issue would be greatly appreciated.
UPDATE 1: Moved workflow manager to own dedicated server and removed WFE from workflow farm. New server instantly ramped up to 99% CPU usage. Event view now shows this warning several times a second:
EventData
location DispatchLoopInstanceAsyncResult.Isolate exception Microsoft.Activities.Dispatcher.WorkflowSessionConfigurationFailedException ---> System.ArgumentNullException: Value cannot be null. Parameter name: workflowDefinition at System.Activities.Hosting.WorkflowInstance..ctor(Activity workflowDefinition, WorkflowIdentity definitionIdentity) at Microsoft.Activities.Hosting.HostedWorkflowInstance..ctor(IHostedWorkflowInstanceOwner owner, HostConfiguration configuration, Activity rootActivity, Guid workflowInstanceId, EventTraceActivity traceId) at Microsoft.Activities.Hosting.WorkflowSession.ConfigureEnvironment(Activity workflowDefinition, IWorkflowConfiguration workflowConfiguration, IEnumerable`1 hostExtensions) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ConfigureWorkflowSessionEnvironment(MessageSessionContext messageSession, WorkflowSession workflowSession, Activity rootActivity, IWorkflowConfiguration workflowConfiguration) --- End of inner exception stack trace --- at Microsoft.Workflow.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SessionPersisted(IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)
Any ideas on this?