I have been able to resolve my issue, I was also facing the same issue and saw this forum, but did not find an answer. I found an answer and thought it might help someone!
I was actually extending the site as HTTPS, so I thought may be the issue was that. So I deleted that site and extended it as a normal HTTP site with FBA authentication. Now I saw the login page, but when I gave user name/pwd it did not sign in. I realized that the FBA user was not having access, at least one FBA user, maybe the admin, etc., has to be set as owner on the SiteCollection. My owners were all NT guys, none from the FBA.
- So, I changed the NT site to allow FBA access as well (So I can find FBA admin1 to give him access)
- Then I logged on the site as windows authentication/NT guy and added one FBA user admin1 as owner (you can add in any group, I was just testing)
- Then I disabled FBA on NT site (Because I don't need internal users to see the Options of logging as NT or FBA)
- Now when I open the extended FBA site, it gives me a login prompt and admin1 can log in! Viola!!
Note: I know there might be a question why the login Page does not come up even without admin1 having access as authentication is secondary, but I don't know. May be cause the site only had NT guys with access, since I chose FBA, it did not find any reason to submit the login page? Since all users were only integrated authentication guys, and I chose not to use integrated authentication, it just directly says "401 Unauthorized"!