Is there a way to restrict the out of the box document libraries like "Document Library" "Picture Library" etc and just have custom document libraries being shown in the pop-up below?
Is your requirement to hide the action or prevent users from creating new lists in the SharePoint site? You could prevent users from creating lists by making them a contributer or create a custom permission level.
You can use the SPRoleDefinition class to programmatically create a permission level. Take a look at
Edit: There is another option. That is to unregister the document library list template from the site. You can do that by disabling the feature: