You can configure FBA on the same web application in SharePoint 2010 without having to extend it. (Your web application will need to be configured to use Claims-based authentication and not classic...I'm assuming that's already the case).
There are numerous tutorials about the steps required to configure FBA, which generally boil down to:
- Creating the ASP.NET membership and role providers
- Modifying the web.config files for the FBA web application, central administration, and SharePoint's security token service
- Configuring users
- Setting up the appropriate authentication provider in central administration and testing
There is a good tutorial that uses the ASP.NET SQL Membership and Role providers here, but you can substitute your own providers.
Out of curiosity, how would you plan to redirect users based on URL if you don't want to extend the web application? The whole point of mixed mode authentication is that the same web application URL can support multiple authentication providers. This helps when it comes to things like search (i.e., search has to use an NTLM account for crawling, and with the same URLs in place, FBA users with appropriate permissions can more readily access this content).
If you did want to do any special redirection, you would need to create your own custom sign in page and specify it in central administration for the web application where you are configuring authentication providers. An example that does this based on client IP address can be found here.