Here is a minimal HTML file that will make use of SP.UI.ModalDialog:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<link rel="stylesheet" type="text/css" href="/_layouts/15/1033/styles/corev15.css"/>
<script type="text/javascript" src="/_layouts/15/init.js"></script>
<script type="text/javascript" src="/_layouts/15/msajaxbundle.js"></script>
<script type="text/javascript" src="/_layouts/15/1033/sp.res.js"></script>
<script type="text/javascript" src="/_layouts/15/sp.init.js"></script>
<script type="text/javascript" src="/_layouts/15/sp.ui.dialog.js"></script>
<script type="text/javascript" src="/_layouts/15/core.js"></script>
</head>
<body>
<button type=button onclick="openDialog()">click me</button>
<div style="display:none">
<div id="d10">my dialog message</div>
</div>
<script>
function openDialog() {
var e = document.getElementById('d10');
var options = {
title: "My Dialog Title",
width: 400,
height: 300,
html: e
};
SP.UI.ModalDialog.showModalDialog(options);
}
</script>
</body>
</html>
This one will work for SharePoint Online, can't say anything about on-premises but I see no reason why it shouldn't work.
Update to make it a proper answer:
You can copy all the script tags + the core15.css style tag from the source view of a normal SharePoint page to your HTML file. Once that's done you can delete the tags one by one (when it throws an error after removing a tag, add it back) until you get to the bare minimum that is required to render the modal dialog.
For SharePoint Online the code above seems to be the minimum.
UPDATE
In my SharePoint 2013 on-premises environment the code below seems to be the minimum required:
(you might have to adjust the long alphanumeric string behind ScriptResource.axd?d= to the appropriate value in your environment)
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<link rel="stylesheet" type="text/css" href="/_layouts/15/1033/styles/corev15.css"/>
<script type="text/javascript" src="/_layouts/15/init.js"></script>
<script type="text/javascript" src="/ScriptResource.axd?d=UpqI01Xk841N_9CuQ5wMNuq9zFYl9ThRGFK83kTirNaZpHlB_KtqOQsuq_rvbhkEoKzWx2eHmGUEVuRsRgkhBYN6G4P88iQ7E9317tHeeWCm7WNLlm-efogSztGRwHF4ALZSZgIi48GJ_F2nixLvqYT_d6nONKgdtAa-ZCAL7pf2SnblS9VJ1CfY9irxg7Kw0&t=ffffffffeea0dba9"></script>
<script type="text/javascript" src="/_layouts/15/sp.init.js"></script>
<script type="text/javascript" src="/_layouts/15/sp.ui.dialog.js"></script>
<script type="text/javascript" src="/_layouts/15/ScriptResx.ashx?name=sp.res&culture=en-us"></script>
</head>
<body>
<button type=button onclick="openDialog()">click me</button>
<div style="display:none">
<div id="d10">my dialog message</div>
</div>
<script>
function openDialog() {
var e = document.getElementById('d10');
var options = {
title: "My Dialog Title",
width: 400,
height: 300,
html: e
};
SP.UI.ModalDialog.showModalDialog(options);
}
</script>
</body>
</html>
Update 2
And you should change the line
html: e
to
html: e.cloneNode(true)
Otherwise the button will only work properly once since you pass the HTML by reference and will be deleted when you close the modal dialog.