All Users (Windows), Everyone and Everyone Except External Users are the SharePoint out of the box groups which does not have permissions on SharePoint site/lists automatically.
I am not sure if there is any way to permanently remove those from People and Groups. However, showing those groups/users under "People and Groups" section does not mean that they have permissions on the SharePoint site.
You can check the actual granted permissions on the SharePoint site from "Site Permissions" page (generally, with URL like:
The groups shown under "People and Groups" are there just as normal groups and admins could possibly use those in their site but it doesn't mean that they have any access until you add these groups under "Site Permissions" or "List Permissions" section explicitly. So everyone except external users appearing under people in groups does not mean everyone in the world can access your site. It just means that the group is there if you need to use it. If you want to give everyone in your company access to your site, you have to add everyone except external users into the "Site Permissions" section.
- Share a SharePoint site
- Understand groups and permissions on a SharePoint site