Is there a way to redirect to a custom site page after clicking the New button in my library?
After clicking, should redirect me to a custom page such as:
SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. It only takes a minute to sign up.Sign up to join this community
We cannot achieve your requirement directly.
There is a workaround for the issue, you could customize the new form by PowerApps. And you can customize the new form according to the custom site page.