Sharepoint 2007 - override anonymous user permissions, version history button issue

Question

I have enabled "restricted access" to a document library so that the "version history" button doesn't appear for non-anonymous users in a specific group, this is working fine. However if I enable anonymous access on the entire website the "version history" button reappears. How can I modify anonymous user permissions such that it has the "restricted access" permission therefore ensuring that the "version history" doesn't appear for both anonymous users and non-anonymous users in a specific group.