You should be able to allow for anonymous users at the web application level. Once you've done that, you should then be able to grant them access at the site collection level. I would create a security group for them, even if you're only going to have that anonymous users in the group for scalability reasons (additionally, I don't think you can just add a user without granting them some level of permission, but you can do this with a group). Don't grant it any access at the site collection level because if you do, you will allow it that access by default to everything on the site.
Once you've created the group, the tricky bit is to do the following:
- Drive to the library you want to grant access to and open up the Library tab in the ribbon.
- Click on the Manage Permissions button.
- Disinherit permissions from the list for the library.
- Grant "Read" permission for that group that you created.
That should be all that you need to do. Of course, if you do do it this way, your users won't be able to access anything on the site except for the URL of the library. If they go to the main page they'll get the Access Denied error. I suggest creating a landing page for them and granting that page anonymous access permission using the same steps noted above.