It's possible and the permission can only be granted to the folder you desire. This would be done by breaking the permission inheritance from the above level and assigning unique permission for the folder in question. However, for avoiding
- unnecessary end-user confusions with multiple "Sorry, you don't have access to this page" when the user attempts to move anywhere around the folder,
- possible layout issues with how the folder is displayed for the user, and
- permission management headache,
you should not do this.
Instead, plan and implement a structure for your document management so that the users always have a logical way to navigate upwards (i.e. from folder to parent library, from parent library to the site, etc.).
Do I need to give him permission on site and document library as well?
You don't have to, but you should.