Based on this, numerous customization options are not supported in the modern list/library like jslink field and view customization, custom css via alternate css url, custom masterpages, custom js embedded via User custom action etc.
The solution to your problem is to use SharePoint framework extensions (SPFx extensions).
SPFx extensions allow you to extend and customize SharePoint experience in modern sites, pages, lists and libraries using client-side tech stack.
There are 3 types of extensions - namely,
ApplicationCustomizers , FieldCustomizers and CommandSets.
For your particular case, you would be using the extension type
CommandSets as it would allow you open a modal dialog on click on a ribbon button.
This is a whole new model to SharePoint development which complements the existing addin model.
For that, you would need to setup the entire dev environment and then deploy the extension.
Reference links below:
Set up your SPFx development environment
SPFx extensions overview
Build your first ListView Command Set Extension and make use of the dialog API to show content in dialog
If you follow the above links, you will end up with the below result:
Sample SPFx extensions links which implement the Commandsets:
That being said, if you want a single solution to work, you should control the list experience via tenant settings as below. You can keep classic to use the old way for all lists or switch to modern. Do note, the classic is neither deprecated nor is it going away:
You can go to SharePoint admin center(
tenant-admin.sharepoint.com) > settings and click on Classic or modern.