Is there a way to redirect to a custom site page after clicking the New button in my library?
After clicking, should redirect me to a custom page such as:
We cannot achieve your requirement directly.
There is a workaround for the issue, you could customize the new form by PowerApps. And you can customize the new form according to the custom site page.