1

I have an over highly customised SharePoint Online system. This system consists of various workflows that help a client company represent their day to day business process. One of the workflows consists of calling 3 web service requests. The Web Service is a WCF service and is coded in C#. It makes use of the CSOM library and does complicated tasks that would normally not be done in SharePoint Designer. These calls can take from 30 seconds up to 3 minutes, to return a value when run in a normal state.

I have run into an issue, with having users initiate a workflow on as much as 20-30 document sets (list items) in a short time frame, only to find out that after some load on the system, the workflows become stuck. Typically, the first few are completed OK, and then problems start to occur. There are 3 web service calls within the workflow, lets call them A, B, C. A and B should take around 10 seconds to return a value. C should take anywhere from 1 to 2 minutes, as it is a bit more intensive and requires some querying on a large library (around 6000 list items).

I have logged exceptions of Web Service C. I have also enabled logging in Web Service A and B to get further information once the same issue occurs again.

The most occurring error is this:

Not well formatted JSON stream.

These are other exceptions that popped up quite a few times:

Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

The underlying connection was closed: An unexpected error occurred on a receive.

The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

These exceptions appeared only a few times:

Version conflict.

The operation has timed out.

Unable to read data. The end of input stream is encountered.

I think that this is an issue with an overload of either SharePoint Online or the Workflow Service of SharePoint Online, or else, the Web Service becoming too busy with requests. I have analysed as well as I could and noticed these things:

  1. I have noticed that for the first 10 problematic workflows, the process got past web service requests A and B, in each case, and then during the execution of C, started returning errors as shown above. For some reason, the same request seems to in some cases have been restarted (I guess from SharePoint) as many as 5 times, sometimes seconds apart and sometimes minutes. At the end, all requests seem to have resolved and returned a true value, and the workflows were completed successfully.

  2. While the previous point was resolving on its own, another 12 workflows were initiated (at the same time), and all started failing on web service request B i.e. earlier within the workflow's logic steps. I do not have exception logs, unfortunately, but I have reconfigured logging for that method and will update once I do. SharePoint then tries to restart the problematic workflow, but still keeps getting stuck. The only resolution is to terminate every workflow manually.

  3. I have noticed that in a separate console application running every 5 minutes (timer job), when running at the same time as the issue, jobs that normally take 2 minutes to complete successfully started taking as much as 14 minutes! And in some cases failing and returning the error mentioned above (which I found practically nothing relevant about): Not well formatted JSON stream. In this case, it is not restarted once failed, and can therefore not be resolved after a few tries. Could this mean that this error actually means that SharePoint is busy and has nothing to do with the connection between SharePoint and the HTTP Web Service?

  4. I have noticed that when a workflow gets stuck on a web service call, SharePoint retries in a few minutes, which in my opinion only contributes to making things worse. And the situation is resolved back to normal, only when accessing every problematic workflow and terminating the workflow from the settings (thank god for well structured views that help me locate issues easily!).

  5. I have noticed that while resolving the issue in 2), I manually tested the WCF service with web service request B, and got a response in as little as 10 seconds. This could have been a result of having already eased the load on the web service at that point (I had already stopped 75% of the problematic workflows at that point) or, due to the fact that the issue was not coming from the web service but rather from SharePoint not liking the fact that more than 20 workflows executing "intensive logic", were simultaneously running.

    Any ideas where to even start from? Apologies for being long but I though only a detailed explanation could enlighten anyone to offer any guidance, whatsoever.

TLDR? - Complicated business process in SPO environment. Workflow running several web service requests is started excessively, on around 30 list items in a short time span. First few are OK. Next 10 start getting issues on the last web service request and exceptions (shown above) are returned intermittently until all 10 are resolved on their own. Next 12 are started (while previous 10 are still resolving), workflows start failing intermittently earlier within the workflow logic steps, on a supposedly quicker web service request (don't have exceptions to show for this yet but will hopefully soon have) and issue is not resolved automatically but requires manual termination. Question is: is it SharePoint being overloaded, is it the Web Service, or a combination of both?

Improve this question
7
  • too much asynchronicity and retrying. I'd rethink all of that processing. Web services doesn't care about states, you have to handle that on the response. It sounds like you need to revamp what you are doing in web services. Maybe a refactoring of the workflow that is in play right now. Sounds like it needs to start one at a time or develop a queuing system, for starts. Having multiple web services run here and there sounds like a nightmare. What do these web services do?
    – Mike
    Sep 28, 2017 at 17:49
  • Your workflow is stepping on itself. @Mike is right, you need to rethink your architecture.
    – Derek Gusoff
    Sep 28, 2017 at 17:54
  • I would also look at optimization of the 3rd call which queries 6000 items of a list. I would suggest that you use index certain columns used and view fields in caml query to reduce the duration. If you can share some code, would be great and we can help you optimize it
    – Gautam Sheth
    Sep 28, 2017 at 18:40
  • @Mike. Thanks for your answer. These web service methods do complex things. A) extract lookup value in item in workflow, query another list to retrieve related item and based on the value of a field in that list item, return true or false. B) based on result of A, write into a database on the same server to schedule information for another application that runs by a timer C) move documents from one location to another, copying metadata in the process. This is a highly customised system, which the client company uses as a substitute to their daily business process.
    – Jurgen Cuschieri
    Sep 28, 2017 at 21:59
  • I will start by redesigning this problematic instance in a way that executes A,B and C in one web service method. I did not simply go for this option before, as there is often 2010 workflows called in between the different actions, and 2010 workflows cannot be called from C# CSOM! But it'd be worth a shot to create the 2010 workflow logic in C# and try it out! So guys, do you think the issue is with the Web Service or with the workflows? After I posted this question I found out somewhere that Sharepoint has a 15 concurrent workflow limit. Couldn't that possibly be the problem?
    – Jurgen Cuschieri
    Sep 28, 2017 at 22:04

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.