I had the same problem when creating my custom login page, and the answer was to add the "/_layouts" to the front of the path I was linking to.
Creating a Simple Shared Login Screen
If you want some form of shared login screen, without needing the IP address or a feature or anything else, here's what I did to get that:
For each site involved, created a new copy of the FBA login folder in LAYOUTS. These folders can then be modified per site and used as the custom login page for that site. (see example)
On my custom login screens, I added a link such as "Employee Login" that points to the NTLM login page (/_layouts/windows/default.aspx)
Had a policy pushed out so that all company machines had the site domains added to their 'Trusted Sites' zone in the browser
The end result is that everyone gets the same login screen. FBA accounts can enter their information normally, and AD users can click the "Employee Login" link to authenticate via NTLM. With the site being trusted, the browser passes their Windows credentials so they log right in with no prompt.
Example of Creating the Custom Login Folder/Page
It's been a couple of years but coming back to this question I thought I'd add the examples from the comments in case those get removed at some point.
Here are the steps & paths involved in creating that custom login page:
- Access your SharePoint server(s). Any WFE or Application server will do, as long as you make the same changes on any WFE serving content to users in the farm.
- Make a copy of folder "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\TEMPLATE\IDENTITYMODEL\FORMS". This is what your webapp uses when accessing the virtual "_forms" directory.
- Paste that folder into "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\TEMPLATE\LAYOUTS". This is the location of the virtual "_layouts" directory.
- Change the name of your pasted folder to something else (i.e. - "CustomLogin" instead of "FORMS". This name will be used in the URL for your new login page.
- If customizing the login page in SharePoint Designer or NotePad++ (or whatever your preferred editor is), you'll also want to look for a line like the one shown below. This is the master page reference. Copy the referenced file into your custom folder and update the referenced path accordingly.
Page Language="C#"
Inherits="Microsoft.SharePoint.IdentityModel.Pages.FormsSignInPage"
MasterPageFile="~/_layouts/15/errorv15.master"
Once you've copied the master page and updated that reference, look further down the page code and you'll see the username/password fields for claims credentials. The two fields will have "SharePoint:EncodedLiteral" in them. The login button is a standard ASP-style button following the password. In this example, we're adding an "Employee Login" link right under that for our internal users.
Beneath the existing login button, add a new row containing a link like so. Alternatively, you can add a button that changes the HREF location when clicked. The button looks a little nicer.
href="http://mysite.server.com/_windows/default.aspx?ReturnUrl=%2f_layouts%2fAuthenticate.aspx">Employee
Login
- Save the page and test it in your browser by manually entering the new custom URL and page name. Confirm it looks ok and that both logins work.
- Update the default login page for your authentication provider, pointing it to the new custom page URL you just tested in your browser.
- If needed, copy the new custom folder to the LAYOUTS location on each server in the farm and/or serving content for the site.
Note: This example is using SharePoint 2013 on an Enterprise farm setup.
The Importance of Customization
I also cannot stress enough that you should customize the login screen once you're done, especially on anything a customer might see. The instructions below will give you the default forms login screen, which is bland and kind of ugly:

A few tweaks to the format of the page and/or the attached master page can make it look like this:

You also may have an issue with the "Employee Login" link dropping off the SOURCE parameter from the URL, which ends up with the employee being redirected back to the root page instead of whatever their destination was. This simple javascript should fix that by grabbing everything after the "authenticate.aspx" and then re-adding it to the URL as they click the button.
<script>
function employeeLogin(){
var winURL = window.location.href;
var dPos = winURL.indexOf("Authenticate.aspx");
var dest = winURL.substring(dPos + 17);
var newURL = "http://mysite.server.com/_windows/default.aspx?ReturnUrl=%2f_layouts%2fAuthenticate.aspx" + dest;
window.location.href = newURL;
}
</script>
Simply call that function when clicking the button, and voila!