Would this not work? Using the fact that Layouts is shared through the fram
targetUrl = '/_layouts/MyProject/AppPage.aspx' + "?Location=" + window.location.href;
or you could try:
targetUrl = SP.Utilities.Utility.getLayoutsPageUrl('MyProject/AppPage.aspx' + "?Location=" + window.location.href)