Well I don't know what TMG is, and there are countless things it could be. I know this question is pretty old, but I'll give it a shot because these kinds of questions come up a lot and deserve an answer - especially since I just figured out some fixes for these kinds of issues & would like to pass them along.
The answer from
In my case, the problem was caused by a custom IIS Http Module, who
was doing some checking on request form variables. It was only
checking if HttpRequest.Form property <> null, and it seems this type
of check crashes the request made by Sharepoint Designer.
Is this what TMG is, a custom IIS Http Module? Don't use that trash!
If it's not an IIS Http Module, or disabling/removing it doesn't work, for any authentication issue, you need to go to IIS on the server and go to the web application, select Authentication. Ensure you have Windows Authentication enabled, the rest disabled. If you have to make changes, ensure you recycle the Application Pool associated with the Web Application (look in the properties of either one to discover what is associated with what for your environment).
Then, try running SPD with the account that is the identity account for the application pool using a
runas /user:domain\user "C:\Program Files..." and the path for SPD. Having to put in the password for that account will ensure that the account is not locked/password hasn't expired. If so, reset it in Active Directory.
If you still have issues after that, it's possible the account doesn't have rights as a site collection admin on the site. Fix this in Central Admin by going to Application Management > Change site collection administrators and select the site collection you're trying to access.
While there, making sure SPD is enabled in Central Admin doesn't hurt: Check/refresh the setting under General Application Settings > SharePoint Designer > Enable managing of the Web Site URL Structure.
Next, try taking off SSL/HTTP and just making it a port 80 binding. Do you still get the issue? If not, it's a certificate issue. Ensure it is imported into the Trusted Root Certificates Store using Run > mmc, File > Add/Remove Snap-in, choose Local Computer, add the Certificates (Local Computer) snap-in, choose Next on the rest of the prompts, Finish. Navigate in the tree, find "Trusted Root Certificates Store", right-click, select Import Certificate. Navigate to the SSL.cer file, select it, select OK, import should be successful. Re-apply SSL/port 443. Recycle the app pool, then try your site again. If you get any certificate errors, be sure whether or not they are truly locking you out or if they just say something dumb like warning you about going from HTTP to HTTPS, or are saying the root certificate is not installed (which is a separate certificate you need in your Trusted Root Certificate Store, also, that is the certificate of the certificate server that authorized your web site's certificate). If none of that works, try getting and applying a new SSL cert.
The specific error in SPD would've been nice to have.