Do you need to retain user access for permission trimming? If not, running elevated could be the quick and easy way to get around the problem. Remember that you'll need to open new instances of site/web inside the block as existing instances will still contain the current user's permission map.
If you don't want to do this, take a look at your code. Giving access to the document library will allow code running on a limited users behalf to access the list using the list GUID or URL/name directly. If you try to get to the list using the web that was opened under their permission, you will get access denied. Referencing /anything/ the user could not see from their browser will get access denied errors.
If the code needs to reference values from another list on which the user has no permission, you either have to run elevated, re-open the web and access the list or grant the user read access to that additional list.
If you can specify some details about what your code is touching that would be helpful.