There is no OOB setting in SharePoint for this requirement. If you want to change the open behavior for the “Attach File” button in the list, you will need to use custom solution, which will be rather complicated since the item needs to be selected first.

Only top voted, non community-wiki answers of a minimum length are eligible