"Modern" list and libraries do not support as many customization options as "classic" lists and libraries. In this article we'll provide details and examples of the supported options. The SharePoint team is working to support more options in the future. The list below gives a quick overview of the supported capabilities for "modern" lists and libraries:
- Subset of User Custom Actions(For your requirement you can use this option)
- Custom branding
- PowerApps and Flow integration
There are numerous customizations which currently are not supported for "modern" lists and libraries:
- JSLink based field customizations - See note on SharePoint Framework Extensions
- JSLink based view customizations - See note on SharePoint Framework Extensions
- Custom CSS via AlternateCSSUrl web property
- Custom master pages - More extensive branding will be supported later using alternative options
- Customization via InfoPath
- Minimal Download Strategy (MDS)
- SharePoint Server Publishing
Reference : Customizing "modern" lists and libraries
At present you cannot add any custom actions in new experience of SharePoint. Microsoft has blocked modifications in SharePoint's new UI.
You can refer this uservoice. Also you can vote it.