It's not possible to do this, as the setting will apply at organization-wide (tenant) level, not user level. The screenshot says it all.
Conditional Access
As for Conditional Access, I'll try my best to answer as I'm not very much expert in it. Indeed, that the setting will create a conditional access policy in Azure AD.

Talking about Conditional Access policy, it's another universe altogether, you could restrict on which groups can access SharePoint Online with specific type of devices, via browser, location, whether there's sign-in risk level imposed, and what client apps they're accessing.

But this will apply to ALL SharePoint Sites regardless.
As you can't granularly control the settings up to the site level, at least not directly/not so straightforward, in 2021 Microsoft introduces Authentication Context where the way apps authenticated can be separated based on different context.
Imagine your scenario, where you want external users access is limited, for a particular SharePoint site. You could set a site to have a different Authentication Context and sensitivity label, then attach this to the Conditional Access policy.

Another link you might want to read up: https://alberthoitingh.com/2021/07/01/authentication-context-and-sharepoint-online/, https://learn.microsoft.com/en-us/sharepoint/authentication-context-example.
I once tried this out with no success so far, time constraint, probably due to the policy sync between Security & Compliance node, SharePoint, and Azure AD Conditional Access policy, and my lack of knowledge in Security aspects of Office 365 (like I said earlier, it's another universe).
Apologies if this doesn't answer your question. If you register a M365 Developer tenant, you can do trial & error on your own time.