I solved the problem by this answer in stackoverflow, provided by Kent Lynch. I copy & paste the answer to here for future reference:
You need to create an sub-directory that acts as a buffer to block/remove the inherited items from the .net 2 / 3.5 framework and then create your application under this.
Assuming you name the buffer application apps, and your custom .NET 4.0 application is called myapp, your resulting application would reside at:
http://[sharepoint-site]/apps/myapp/
How to do this:
Create a sub-directory apps under the root of your SharePoint site
Go into security for the apps directory and add everyone with Read permissions
In IIS, convert this to an application and pick the same app-pool
that your SharePoint site is running under
Create a web.config under /apps/, this will block/remove the SharePoint stuff (see below for the code block)
<configuration>
<system.web>
<httpHandlers>
<remove path="Reserved.ReportViewerWebControl.axd" verb="*" />
</httpHandlers>
<httpModules>
<clear/>
</httpModules>
<webParts>
<transformers>
<clear />
</transformers>
</webParts>
</system.web>
<system.webServer>
<handlers>
<remove name="OwssvrHandler" />
<remove name="ScriptHandlerFactory" />
<remove name="svc-Integrated" />
<remove name="ScriptHandlerFactoryAppServices" />
<remove name="ScriptResource" />
<remove name="JSONHandlerFactory" />
<remove name="ReportViewerWebPart" />
<remove name="ReportViewerWebControl" />
</handlers>
<modules>
<!-- depending on any customizations, you may have to add/remove items from this list as needed -->
<remove name="SPRequestModule" />
<remove name="ScriptModule" />
<remove name="SharePoint14Module" />
<remove name="StateServiceModule" />
<remove name="PublishingHttpModule" />
<remove name="RSRedirectModule" />
</modules>
<httpErrors errorMode="Detailed"></httpErrors>
</system.webServer>
</configuration>
- Create your myapp directory under apps (ex. /apps/myapp/)
- In IIS, go into Application Pools, create a new AppPool, MyApp .NET v4.0
- Go into Advanced Settings > Identity and add the same AD domain user account credentials that your SharePoint site is using
- Still in IIS, go back to myapp and convert to an application and pick the MyApp .NET v4.0 AppPool