Using ListView Command Set extension:
If you want the new command in top bar you have to develop a SPFx ListView Command Set extension.
You can find more information at: Build your first ListView Command Set extension
Using Column formatting:
If there is no requirement to have the link/button in top bar only, you can create a link using SharePoint JSON column formatting capabilities. This is an easiest way without SPFx development.
Use below JSON in Format this column option:
"txtContent": "Click here to open link",
You can use this JSON on any supported column in document library, few suggestions:
- Use default
Title column if you are not using it for storing specific document titles.
- Create a calculated column with formula
=[Title] & use above JSON on this new column
- Create a Single line of text column --> Add it to library view --> Hide it from library forms/properties form (optional).
Documentation: Use column formatting to customize SharePoint