Document library custom button in ribbon

Question

Is there a way to add a custom button inside the ribbon on the modern ui? - but I cannot modify the master page in any way shape or form

Answer 1

"Modern" list and libraries do not support as many customization options as "classic" lists and libraries. In this article we'll provide details and examples of the supported options. The SharePoint team is working to support more options in the future. The list below gives a quick overview of the supported capabilities for "modern" lists and libraries:

• Subset of User Custom Actions(For your requirement you can use this option)
• Custom branding
• PowerApps and Flow integration

There are numerous customizations which currently are not supported for "modern" lists and libraries:

• JSLink based field customizations - See note on SharePoint Framework Extensions
• JSLink based view customizations - See note on SharePoint Framework Extensions
• Custom CSS via AlternateCSSUrl web property
• Custom JavaScript embedded via User Custom Actions - See note on SharePoint Framework Extensions
• Custom master pages - More extensive branding will be supported later using alternative options
• Customization via InfoPath
• Minimal Download Strategy (MDS)
• SharePoint Server Publishing

Reference : Customizing "modern" lists and libraries

Answer 2

At present you cannot add any custom actions in new experience of SharePoint. Microsoft has blocked modifications in SharePoint's new UI.

You can refer this uservoice. Also you can vote it.