No, you are doing nothing wrong.
The groups you created are site wide groups, they are not limited to one Sharepoint list. So when you add a user to any group and this group has permission "Read" for your list, this user will receive read permission for your list. The user will also be added to the general group, so if this group has site-wide read access your user will have site wide read access.
Let's go with an example:
MySiteCollection has following Groups: "Special Reader", "Special Admins".
- Special Readers have read access on MySiteCollection
- Special Admins have full access on MySiteCollection
MyList has unique permissions, not inheriting from MySiteCollection
- Special Reader have NO access
- Special Admins have contribute access
In this scenario when you would add "Mark Jones" to the group "Special Readers", he would have full read access on MySiteCollection, but he would not be able to read MyList.
So basically when you click "stop inhering permissions" you can set special permissions for existing groups/users, you do not create new groups.
PS: However you could of course another group, only including users for MyList with special permissions for MyList and no permissions for MySiteCollection.