System.StackOverflowException at Microsoft.Xslt.NativeMethod.CheckForSufficientStack

Question

Error while executing web part: System.StackOverflowException: Operation caused a stack overflow. at Microsoft.Xslt.NativeMethod.CheckForSufficientStack

I got the above error when using the OOTB search functionality in the page. OSSSearchResults.aspx

I noticed this because there was an error on the webpart

Unable to display this Web Part. To troubleshoot the problem, open this Web page in a Microsoft SharePoint Foundation-compatible HTML editor such as Microsoft SharePoint Designer. If the problem persists, contact your Web server administrator.

Correlation ID:41e19279-3151-4ac7-b12f-c396c73a7599

With the correlation id, I could find the problem in the log.

Answer 1

Despite the Exception thrown the problem is a XSLT Extension method taking to long time to complete.
In SP2010 SP1 the SharePoint team introduced a check in the XsltTransformations done by Data Form WebPart and thereby in Search Core Results WebPart (as it inherit from that), that the latest entension method has to complete within 1 second of the first being invoked otherwise they throw the StackOverflowException, this exception was probably choosen because it could already be thrown by the XsltTransformation, so code checking for exceptions were likely to catch this. (Prior to SP1 the limit was 5 seconds)

In the SP2010 February 2012 CU a new property has been added to the SPFarm to allow you to set the timeout limit. See XSLT and timeout problem when transforming runs more than one second