hence why logged in users have access and can use the JS file whereas annonymous users dont have privlages ;) so it doesnt work!
This means that you need impersonation! but that requires cutomisation of webservices and other things like webparts ect where you can put in code to impersonate ;)
my personal sugestion is to create a feature that adds the JS to the page rather than it being in the masterpage! You can then set impersonation for the reference for it to work for annoymous users ;), have the feature scoped to site level!
bingo! you have JS files in _Layouts folder enabled for the annoymous users!
for referencing the JS in feature:
what type of master page are you using? you should look into publishing masterpage that should contain the correct access for annoymous users using JS
So you now have your reference in the master page to the JS files in 12/14 hive within the layouts folder!
to grant access you need to ammend your web.config file like so:
<allow users="*" />
the above code is giving access to all users (including anonymous users) to the JS file. So you can keep the current masterpage ;)