If you mean that the end user should not be able to write www.youdomain.com in any browser and see the result?
No that is not possible. The request and the rendering in the IFrame is the same as if the user had opened a new tab.
I believe external users wont have access to your Azure AD and hence this issue (I would say security feature).
Why would you allow external users to see the list of users in your organization. (This is a security issue as it would expose all email IDs resulting in spam/phishing emails if data is not protected)