SharePoint supports a very wide variety of web services that can all be called via AJAX. However, those services are all called using the ID of the user, which is sometimes not desired. In those instances, you need to use a solution like it sounds like you are using already since pages running on the server can run either in the User context or in the context of the Application Pool. They can even run be configured to run under a completely different account if that is needed.
AJAX itself runs only on the client and doesn't know anything at all about the master page or where the master page resides. What you can do is put your ASPX page in _layouts then use javascript to determine the current site URL, add '/_layouts/AppName/CustomPage.aspx' to the URL and make your call. This works because '_layouts' is the same no matter which site you are in but calling it in this way allows your ASPX page to easily determine the current site context and respond accordingly.