Why are you deploying your Custom Web Service to the layouts folder at first place?
You need to copy the Web service files to the _vti_bin virtual directory
Copy the Service1wsdl.aspx file, the Service1disco.aspx file, and the Service1.asmx file to the _vti_bin virtual directory. This is the directory where all default Web services are stored.
Copy the corresponding assembly (.dll) file to the _vti_bin/bin virtual directory.
Note The _vti_bin/bin virtual directory is mapped to the \Server_Name\Program Files\Common Files\Microsoft Shared\Web Server Extensions\ISAPI directory.
Next, you must include the service in the list of default Windows SharePoint Services Web services that is displayed in the Add Web Reference browser of Visual Studio .NET. Follow the complete tutorial here
For setting the anonymous access to the Web service: Go to IIS -> Select the Web site -> Go to _vti_bin folder -> Select the custom web service -> Right click and enable the anonymous access.
Update: Just adding a very helpful tool for developing the custom web service