Here is the permission settings of my user:

 - Site: Limited access
 - Library: Limited access
 - Document Set (inside library): Full Control.

**Issue**: When accessing directly the document library, the user is unable to upload documents in the document library. He can see the documents but all the controls are disabled. Now If I give 'Read/View Only' permission at sub-site level, he is able to upload documents in the document set.

'Read/View Only' permission at site or library level is not an ideal solution for me.

What am I missing? 
Why unique permission is not working?