We have two 'identical' SharePoint farms; one production and one test. Both consist of 4 VMs with identical specifications. An application server, 2 WFEs and a SQL server. Both environments run a single web application which has the same WSP installed. The test environment has been restored from a site collection backup of the production environment so the content in both environments is for all intents and purposes the same. I have checked the load on all servers in both farms and resource usage (memory, disk, processor, network) is consistent for matching servers across both farms.
We have a publishing page on a web in the site collection which has a single web part which renders a custom view of an item from a list in the parent web that contains around 8500 items. The item is retrieved using Linq2SharePoint by the integer Id assigned by SharePoint when the item is created. In the production environment the page consistently takes around 10 seconds longer to load than on the test environment.
All other pages have roughly the same load times across the two environments. Other pages which perform the same function, using the same web part, but in different sub webs in the site collection have roughly the same load times. The performance difference is very specific to this one location where the web part is used.
I think I know where the delay is happening but have no idea why or how to resolve it. Studying the ULS logs and the trace output for the page in the production environment I can see the following line (obviously the execution time varies slightly between requests but is more or less consistent)
Leaving Monitored Scope (EnsureListItemsData#1). Execution Time=11006.7259945049
In the test environment the execution time is around 700ms. This seems a clear candidate for the approximately 10 second performance difference.
My question then is what is going on here? What happens in this method? Why is it taking so much longer in seemingly 'identical' environments? What can I do to resolve the problem?