The central Administration is a web application that has its own alternate access mapping settings.
So you can edit the alternate access for this web application that will not affect other web application AAM settings.
In your case, you can't open central administration to edit these settings now, so you should use Powershell as below
Remove-SPAlternateURL -Identity https://InternalURL - Zone 'Zone Type'
Note: you should set the Zone type based on the zone that you have provided during configuring AAM like 'Internet' or 'Custom' or 'Intranet' 'Extranet'. Check also Remove-SPAlternateURL cmdlet, New-SPAlternateURL
As I have mentioned in the comments, try to get all AAM URLs for Central Administration URL in all zone using the below command line:
Get-SPAlternateURL -WebApplication "CA URL"
Based on the result list of the above cmdlet, remove the URL as below
Remove-SPAlternateURL -Identity http://Incoming URL
Repeat the above cmdlet for all internal URLs (incoming URL) that you have added to the default zone.
Consider you have added an internal link (incoming URL) "https://test" that should have a
default zone as shone below
So you should delete this entry as below
Remove-SPAlternateURL -Identity https://test
Note: You cannot delete the default zone for the main URL