I had faced the similar issue when i was working with STS. The reason you get the error is because there is no trust being established between the Sharepoint and STS .
If i am not wrong you might be getting the error at the below lines of code :) This is definitely a certificate issue.
SecurityTokenService sts = new CustomSecurityTokenService(CustomSecurityTokenServiceConfiguration.Current);
SignInResponseMessage responseMessage = FederatedPassiveSecurityTokenServiceOperations.ProcessSignInRequest(requestMessage, User, sts);
Make sure the app pool account which is running the STS code had permissions to access the certificates (STSTestCert) .
The Identity of the App Pool the Custom STS is running under .Give the Identity read access to the private key of the certifcate used to encypt the SAML token.
The permissions can be added using the certificate MMC snap-in.
2.If doesnt work the STSTestCert might be corrupt or not accessible .Follow http://discusscode.blogspot.com/2012/01/keyset-does-not-exist-issue-in-creating.html, delete the trust and again try adding the AuthProvider using powershell script.