The use of the JavaScript Object Model is not supported in modern views. (Reference)
If you want to show a dialog using SPFx, @microsoft/sp-dialog is what you should be using.
Here are the guidance docs for using that: https://docs.microsoft.com/en-us/sharepoint/dev/spfx/extensions/guidance/using-custom-dialogs-with-spfx