In my experience 100 groups is not unusually high and well within the boundaries defined by Microsoft.
The boundaries pertinent to the this topic are as follows:
10,000 SharePoint groups per site collection
5,000 Active Directory Principles/Users in a SharePoint group (...depending on how many users you have in your org this is where the "all users" idea can fall apart)
1,000 Security scopes per list
Security principal: size of the Security Scope = 5,000 per Access Control List (ACL)
I would say that as long as you're not breaking inheritance within the list either at a folder or individual item level you shouldn't be concerned that 100 groups are going to cause any performance problems.
Link the 2010 capacity management doc if you want more details: http://technet.microsoft.com/en-us/library/cc262787.aspx#SiteCollection