I realize I've been looking in the wrong place entirely.
A quick google search provided me little (useable) information on using the REST API in user context, but this was due to my lack of knowledge with API's in general.
After Bharat's comment I realized that the problem was creating an app, and with some more googling I found the following answer from Lee Liu:
https://stackoverflow.com/questions/50598598/the-oauth2-authentication-request-process-of-office-365-forwards-the-link-to-htt
This is exactly what I needed, I created a Cloud Only user (federated requires some extra steps it seems) and managed to get the digest token in the end.
With the REST API in user context I can follow Bharat's steps to exclude all permissions except the ones in the list.