If your Service Instance is not configured to support MinRole, you must deploy the SharePoint server as Custom. If you have the source or can get the vendor to update it, yes it is possible to make it MinRole compliant.
The article Managing a MinRole Server Farm in SharePoint Server 2016 has information at the bottom about how to make a Service Instance compliant with particular MinRoles.
There is no loss of ZDP when using Custom provided your Service Instances are balanced between 2 or more servers. You can also mix MinRole and Custom within the same farm, e.g. I have 2 Custom FEs because of SSRS and 2 App+Search.
The only thing you lose is Service Instances will not auto start on a Custom role based on provisioning of Service Applications.