1

I have a SharePoint 2013 on premise farm that recently started experiencing high CPU usage on the WFE. The server has been locked at 99% for days on end now and the services that are causing it are all workflow related:

enter image description here

We are not sure what is causing this. We cannot seem to find any workflows that are running off the rails which is the only thing we can think of that would cause this. The workflow farm appears to be in a healthy state and reconfiguring it isn't really an option at this time as that would terminate all running instances of workflow that our company heavily relies on. Our farm consists of 1 WFE, 1 App server, and 1 Database server. Workflow is configured on the WFE and yes we are aware that this is not best practice.

Here are two events that started getting thrown as soon as the high CPU usage began:

ServiceBus threw MessageNotFound exception while trying to receive a deferred message. Deferred message context details: MessageId = 573074, ActionNamespace = Microsoft.Workflow.Management, ActionName = Terminate, InstanceId = {6b339f70-34ce-464a-8b62-e27dea949015}, ResumptionKey = (null), MessageTraceId = {44259071-166f-4d87-b770-d26787e35b85}

/sharepoint/default/81f1359d-754d-4ef9-b013-c63410d5762b/156d509e-da00-4f68-be8e-2ded413ffb90/53a8f467-11ff-ff92-ee48-0070fd375256_dbbeb1cc-941b-4cfb-a778-59bd88be507d: The instance attempted to process the same message too many times. The message may be causing an unexpected failure. The limit is controlled by the MaxMessageProcessingAttemptsOnUnknownFailure and MaxMessageProcessingAttemptsOnKnownFailure system configuration values (currently set to 10 and 1000). The actual values seen are 0 and 1000.

Any insight on this issue would be greatly appreciated.

UPDATE 1: Moved workflow manager to own dedicated server and removed WFE from workflow farm. New server instantly ramped up to 99% CPU usage. Event view now shows this warning several times a second:

  • EventData

    location DispatchLoopInstanceAsyncResult.Isolate exception Microsoft.Activities.Dispatcher.WorkflowSessionConfigurationFailedException ---> System.ArgumentNullException: Value cannot be null. Parameter name: workflowDefinition at System.Activities.Hosting.WorkflowInstance..ctor(Activity workflowDefinition, WorkflowIdentity definitionIdentity) at Microsoft.Activities.Hosting.HostedWorkflowInstance..ctor(IHostedWorkflowInstanceOwner owner, HostConfiguration configuration, Activity rootActivity, Guid workflowInstanceId, EventTraceActivity traceId) at Microsoft.Activities.Hosting.WorkflowSession.ConfigureEnvironment(Activity workflowDefinition, IWorkflowConfiguration workflowConfiguration, IEnumerable`1 hostExtensions) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ConfigureWorkflowSessionEnvironment(MessageSessionContext messageSession, WorkflowSession workflowSession, Activity rootActivity, IWorkflowConfiguration workflowConfiguration) --- End of inner exception stack trace --- at Microsoft.Workflow.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SessionPersisted(IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)

Any ideas on this?

Improve this question

2 Answers 2

Reset to default
0

It's a rush load on work flow manager, and you have to increase the memory in your server to avoid such critical situation

Improve this answer
0

Try to increase the Maximum allowed correlation filter as shown below:

  1. Stop-SBFarm

  2. Run the powershell:

    Set-SBRuntimeSetting -Name MaximumNumberOfCorrelationFiltersPerTopic -Value 1000000

  3. Run the SQL query against SBMessageContainer01:

    UPDATE TopicsTable SET MaxCorrelationFilters = 1000000

  4. Start-SBFarm

After doing this, remember to increase memory on the current Server to take care of the high consumption

In additon, normally we suggest you install workflow at App server rather than WFE server.

Similar issue for your reference:

https://social.msdn.microsoft.com/Forums/en-US/374dcdbf-1022-4d72-9ca5-33ec05d1c712/designer-workflow-statusterminated-the-workflow-exceeded-the-maximum-number-of-attempts-to?forum=sharepointcustomization

SharePoint 2013 Workflow Manager 1.0 Causes High CPU on Server

Improve this answer
1
  • Thanks for the response! I did try increasing the maximum allowed correlation filter as I also came across the article you referenced. This however did not resolve the issue in our environment. I have not increased the memory on the server yet, but the server already has 24GB allocated to it and only appears to actively use 9-10GB at any given time. Could there be a configuration somewhere that is throttling memory usage and thus causing more CPU usage?
    – sisle238
    Jan 6, 2021 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.