It seems that you have created a custom restricted view permission at the site level to work around the issue of your document library not being in the modern experience. However, you are concerned that adding users to this group with restricted view permission may give them access to other lists and libraries visible to guest users.
To address this, you may want to consider creating a new group with unique permissions specifically for the document library in question. In this way, you can grant guest users with contribution privileges to the library while ensuring that they do not have access to other lists and libraries.
Here's how you can create a new group with unique permissions for your document library:
- Go to the site where the document library is located.
- Click on the gear icon in the top right corner and select "Site settings."
- Under the "Users and permissions" section, click on "Site permissions."
- Click on "Create Group" and enter a name for the group (e.g. "Library Guests").
- Set the group's permission level to "Contribute."
- In the "Give group permission to" section, select "List/Library" and choose your document library from the dropdown list.
- Click "Create" to create the new group with unique permissions.
Once you have created the new group, you can add guest users with contribution privileges to the group. These users will only have access to the specific document library and will not be able to view other lists and libraries visible to guest users.
I hope this solution works for you.