To Open the SharePoint Modal dialog in the custom aspx page the following references need to be added.
<SharePoint:ScriptLink ID="ScriptLink1" Name="init.js" runat="server" OnDemand="false" LoadAfterUI="true" Localizable="false" />
<SharePoint:ScriptLink ID="ScriptLink2" Name="sp.init.js" runat="server" OnDemand="false" LoadAfterUI="true" Localizable="false" />
<SharePoint:ScriptLink ID="ScriptLink3" Name="sp.runtime.js" runat="server" OnDemand="false" LoadAfterUI="true" Localizable="false" />
<SharePoint:ScriptLink ID="ScriptLink4" Name="sp.core.js" runat="server" OnDemand="false" LoadAfterUI="true" Localizable="false" />
<SharePoint:ScriptLink ID="ScriptLink5" Name="sp.js" runat="server" OnDemand="false" LoadAfterUI="true" Localizable="false" />
<SharePoint:ScriptLink ID="ScriptLink7" Name="sp.ui.dialog.js" runat="server" OnDemand="false" LoadAfterUI="true" Localizable="false" />
and register the SharePoint.WebControls using below tag in the page
<%@ Register TagPrefix="SharePoint" Namespace="Microsoft.SharePoint.WebControls" Assembly="Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
and please make sure that you've added reference to the MicorsoftAjax.js library in the page to avoid issue with recognizing type of SharePoint ScriptLink
<script type="text/javascript" src="/_layouts/15/MicrosoftAjax.js"></script>