When you call the page you can use the
IsDlg=1 query string. That may be enough for what you need, but it depends on how the custom page was implemented and if it's still make's use of the notdlg classes.
Here is a potential answer on Stack Exchange regarding dynmaicly setting asp.net masterpages:
Just tested this and it works from the PreInit of the Page that is using the nested MasterPage.
protected void Page_PreInit(object sender, EventArgs e)
this.Master.MasterPageFile = "/Site2.Master";
Another method that may be possible is creating your own pagelayout class, that is used in publishing pages. If you look at the pages they don't seem to reference a master page directly, but pagelayout assembly.
Back to the notdlg classes, you could modify the masterpage and add in the MS class to the sections you want to hide and then just utilize the query string method.
To clarify there are actually two different classes depending on version
s4-notdlg - SP 2010
ms-dialogHidden - SP 2013