Take a look here: 10 things I learned about versions
Versions do not disappear until you edit the document or manually
remove the versions.
If you go from "unlimited" major versions to 10 major versions, the
documents in your library will keep their 10+ versions until you edit
the documents or their properties. So that old document, “Production
planning wk. 21, 2011”, with 56 versions, will keep its versions until
you edit the document. But… this document will likely not be edited
anymore, so even if you limit versioning, the versions stay there
forever. I make it a habit of removing all but 3 versions of these documents manually (in the Version History) whenever I find them. I am also thinking about a workflow to do this for all documents in a library.
UPDATE:
Assuming you activated the "Keep the following number of major versions" and set its value to 3 only after documents in document library has been uploaded/edited (creating more than 3 versions), you have to edit the document or manually delete unwanted versions in order to trigger the new configuration and only retain the last 3 versions of any document.