I have an over highly customised SharePoint Online system. This system consists of various workflows that help a client company represent their day to day business process. One of the workflows consists of calling 3 web service requests. The Web Service is a WCF service and is coded in C#. It makes use of the CSOM library and does complicated tasks that would normally not be done in SharePoint Designer. These calls can take from 30 seconds up to 3 minutes, to return a value when run in a normal state.
I have run into an issue, with having users initiate a workflow on as much as 20-30 document sets (list items) in a short time frame, only to find out that after some load on the system, the workflows become stuck. Typically, the first few are completed OK, and then problems start to occur. There are 3 web service calls within the workflow, lets call them A, B, C. A and B should take around 10 seconds to return a value. C should take anywhere from 1 to 2 minutes, as it is a bit more intensive and requires some querying on a large library (around 6000 list items).
I have logged exceptions of Web Service C. I have also enabled logging in Web Service A and B to get further information once the same issue occurs again.
The most occurring error is this:
Not well formatted JSON stream.
These are other exceptions that popped up quite a few times:
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
The underlying connection was closed: An unexpected error occurred on a receive.
The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
These exceptions appeared only a few times:
The operation has timed out.
Unable to read data. The end of input stream is encountered.
I think that this is an issue with an overload of either SharePoint Online or the Workflow Service of SharePoint Online, or else, the Web Service becoming too busy with requests. I have analysed as well as I could and noticed these things:
I have noticed that for the first 10 problematic workflows, the process got past web service requests A and B, in each case, and then during the execution of C, started returning errors as shown above. For some reason, the same request seems to in some cases have been restarted (I guess from SharePoint) as many as 5 times, sometimes seconds apart and sometimes minutes. At the end, all requests seem to have resolved and returned a true value, and the workflows were completed successfully.
While the previous point was resolving on its own, another 12 workflows were initiated (at the same time), and all started failing on web service request B i.e. earlier within the workflow's logic steps. I do not have exception logs, unfortunately, but I have reconfigured logging for that method and will update once I do. SharePoint then tries to restart the problematic workflow, but still keeps getting stuck. The only resolution is to terminate every workflow manually.
I have noticed that in a separate console application running every 5 minutes (timer job), when running at the same time as the issue, jobs that normally take 2 minutes to complete successfully started taking as much as 14 minutes! And in some cases failing and returning the error mentioned above (which I found practically nothing relevant about): Not well formatted JSON stream. In this case, it is not restarted once failed, and can therefore not be resolved after a few tries. Could this mean that this error actually means that SharePoint is busy and has nothing to do with the connection between SharePoint and the HTTP Web Service?
I have noticed that when a workflow gets stuck on a web service call, SharePoint retries in a few minutes, which in my opinion only contributes to making things worse. And the situation is resolved back to normal, only when accessing every problematic workflow and terminating the workflow from the settings (thank god for well structured views that help me locate issues easily!).
I have noticed that while resolving the issue in 2), I manually tested the WCF service with web service request B, and got a response in as little as 10 seconds. This could have been a result of having already eased the load on the web service at that point (I had already stopped 75% of the problematic workflows at that point) or, due to the fact that the issue was not coming from the web service but rather from SharePoint not liking the fact that more than 20 workflows executing "intensive logic", were simultaneously running.
Any ideas where to even start from? Apologies for being long but I though only a detailed explanation could enlighten anyone to offer any guidance, whatsoever.
TLDR? - Complicated business process in SPO environment. Workflow running several web service requests is started excessively, on around 30 list items in a short time span. First few are OK. Next 10 start getting issues on the last web service request and exceptions (shown above) are returned intermittently until all 10 are resolved on their own. Next 12 are started (while previous 10 are still resolving), workflows start failing intermittently earlier within the workflow logic steps, on a supposedly quicker web service request (don't have exceptions to show for this yet but will hopefully soon have) and issue is not resolved automatically but requires manual termination. Question is: is it SharePoint being overloaded, is it the Web Service, or a combination of both?