The use of the JavaScript Object Model is not supported in modern pages. ([Reference][1]) 

If you want to show a dialog using SPFx, [@microsoft/sp-dialog][2] is what you should be using.

Here are the guidance docs for using that: https://docs.microsoft.com/en-us/sharepoint/dev/spfx/extensions/guidance/using-custom-dialogs-with-spfx


  [1]: https://docs.microsoft.com/en-us/sharepoint/dev/sp-add-ins/complete-basic-operations-using-javascript-library-code-in-sharepoint
  [2]: https://docs.microsoft.com/en-us/javascript/api/sp-dialog?view=sp-typescript-latest