How can you add custom permissions in the "Site Permissions" tab just like the standard three in the image below?
I don't think you can customize this SharePoint default panel to add your custom groups/permission levels on it.
However, you can access all your permissions by following below steps:
Click on settings (gear) icon
Click on "Site permissions"
Click on "Advanced permissions settings"
It will take you to the site permissions page.
Alternatively, you can access the site permissions & groups pages using below shortcut URLs:
- Site permissions:
- Site groups: