My list X has unique permissions. Group A has contribute and approve permission on this list, but only read permissions to the whole site. However, when group A tries to launch a workflow manually in list X (via Files -> Workflows) it says: "Sorry, this site hasn't been shared with you."
Giving them full control on the list did not solve the problem, but full control on the whole site worked. So I assume that I have to give group A give permissions somewhere else on the site to allow them to launch workflows. But where would it be?
With SP Designer I found the Document Library "Workflows". I gave group A full control to this library, but it didn't solve my problem.
For future readers: As stated in an old question (linked in the answer below), the behaviour is by design. A user at least needs to have contribute permissions on the site level, in order to launch workflows manually.