Sharing just a library is messy and discouraged by Microsoft. Sharing is supported and easily done for files, folders and sites.
If you want to share a whole library, but not the rest of the site, you need to apply permissions gymnastics that can turn ugly fast.
- Create a new SharePoint group in the site. It should not have any permissions assigned at the site level
- Edit the library, break permissions inheritance and add that new group with read or edit permissions to the library
- Add the external user to the group, which will send them the usual invitation. The invite will contain the title of the site, not the title of the library, which may be confusing to the user.
A much cleaner approach would be to create a separate site that is shared with external users and use the regular way to share the site.