I am really baffled by this issue. I have a site with two collections. For instance example.org/pages/home.aspx and example.org/sites/test/pages/hometest.aspx. For the 1st site anonymous users get a login prompt when trying to access any document or pdf. For the second site I am able to access documents and pdf files with no problem. Both sites fall under the same web application permissions and IIS permissions so there is a setting (I am presuming) at the site collection level that is somehow different between the two. They are publishing sites and both collections have the lockdown site feature enabled. Both sites have anonymous user permissions set correctly. Is there anything else in the site collection settings that would cause this issue.
To access documents anonymously, you need to configure the followings:
1. Enable Anonymous at web application level
2. Enable anonymous at site collection level
If the library has unique permissions, you also need to check the anonymous setting at library level via Library settings->Permissions for this library->Anonymous Access.
After the above, as you are using a publishing site, you may need to deactivate "Limited-access user permission lockdown mode" at site collection features.