Depending on the number of documents you expect your users to store in SharePoint it sounds as if you're heading in the direction of disaster, by storing everything in the same Site in the same Site Collection.
Answer to asked question
You want to create an AD-Group for each department.
You should then add all departments to the guest/visitors Group in the site with all your document libraries. This will give them access to browsing that site.
For each document library you'll then have to :
- Break inheritance of permissions
- Remove permissions for the guest/vistors Group
- Add the appropriate rights for each AD-Group which needs access
To do these things:
- go to the document library.
Library Permissions in ribbon
Stop Ihneriting Permissions
- Select the Groups which shouln't have access and click
Remove User Permissions
Grant Permissions enter the AD-Group(s) and give the group(s) the permissions directly
SharePoint doesn't use the concept of Deny (Except at
Permission Policy at the web application level, which you don't want to/can't use).
SharePoint only uses grant permissions.
By default every securable object (Item/Document, Folder, List/Library, Site, Site Collection) inherits permissions from their parent, but this chain can be broken.