XML for the button is as follows:
<CommandUIExtension>
<CommandUIDefinitions>
<CommandUIDefinition Location="Ribbon.Library.ViewFormat.Controls._children">
<Button Id="Ribbon.Library.ViewFormat.About"
Command="AboutButtonCommand"
LabelText="About"
Image32by32="{SiteUrl}/_layouts/15/1033/Images/formatmap32x32.png?rev=23"
Image32by32Top="-273"
Image32by32Left="-1"
Description="About"
TemplateAlias="o1" />
</CommandUIDefinition>
</CommandUIDefinitions>
<CommandUIHandlers>
<CommandUIHandler
Command="AboutButtonCommand"
CommandAction="javascript:aboutScript({SelectedItemId});"
EnabledScript="javascript:onlyOneItemSelected();" />
</CommandUIHandlers>
As we see in the CommandUIHandler, the CommandAction and the EnabledScript just contain calls to a JavaScript-function. These functions are placed in a simple JavaScript file:
AboutButtonScript.js
function aboutScript(itemId) {
alert("Hello user! You have selected item " + itemId);
}
function onlyOneItemSelected() {
return (SP.ListOperation.Selection.getSelectedItems().length == 1)
}
To make this code available, we will store the file with the JavaScript-code in a document library in our SharePoint site. In the example, I prepared a library “Scripts” to store the file.
Now we can store the script file in the library, add the extension to the ribbon and make the file available for the ribbon extension. This could be done with the Office 365 PnP PowerShell extensions:
Add-PnPFile -Path .\AboutButtonScript.js -Folder "Scripts"
$ribbon = Get-Content .\MyRibbon.xml
$ribbon = [string]$ribbon
Add-PnPCustomAction -Name "RibbonTester" -Title "RibbonTester" -Description "-" -Group "Tester" -Location "CommandUI.Ribbon" -CommandUIExtension $ribbon -RegistrationType ContentType -RegistrationId 0x0101
Add-PnPJavaScriptLink -Name "AboutButtonScript" -Url https://mytenant.sharepoint.com/sites/perm-tester/Scripts/AboutButtonScript.js -Scope Web
I took it from below mentioned article. Kindly follow it so it will be easy for you.
SharePoint – Ribbon-Scripts with external JavaScript-files
UPDATED
According to MSDN that is not possible:
CustomAction cannot contain JavaScript: Any UrlActions or
CommandActions must be a URL to navigate to. The URL can be
parameterized with normal custom actions tokens in addition to the
app-specific tokens.
Using HostWebDialog seem to be the preferred approach for this scenario. You might also consider passing an additional parameter for your page in the App Web, so it will identify it was opened from CustomAction.