I created one site with forms based authentication and anonymous access. That is working fine. Now i want to replace the master page with the custom master page. I just copied the nightandday.master page and pasted in the same directory of the master pages. Now I changed the master page in the site settings page of the site to my custom master page. With custom master page when I signed out it is showing the sign-in page always. I am unable to show the site with anonymous access with my custom master page. I didn't approve my custom master page and I didn't publish it. Is it may be the problem? If not, How to over come this?
Always make sure to check in, publish, and approve your custom masterpages as they will not be visible for anonymous user as they have no permissions to see non-published content.
Have you enable the anonymous access at site collection level?