Here is the permission settings of my user:

Site: Limited access

Library: Limited access

Document Set (inside library): Full Control.

Issue: User when navigate directly to the document set, is unable to upload documents in the document set. He can see the documents but all the controls are disabled. Now If i give Read/view Only permission in the site level, he is able to upload documents in the document set.

 What am i missing? 

Why unique permission is not working? Giving read/view permission at site or library level is not an ideal solution for me.