We need to customize
newdocset.aspx of document set in SharePoint 2010. Currently reference is given to
_layouts folders application pages. Did lot of research, but I got no where.
Microsoft has a sample of how to do this on MSDN. You would basically create an Application Page that serves the same function and then set the "new" url of the document set content type (or your custom one build on it) to that new page:
My team and I have found a custom solution that does not require a new feature or application page.
Using SharePoint Designer, create a new Edit form at the target Library. For this example, we'll call this form CustomNewDocSet.aspx
In SharePoint Designer, navigate to a List and create a New Form of type New item form. Then, take that markup and place it into "CustomDocSet.aspx". This will update the 'Type' of this form from Edit to New. At this point, you associate this form with the target Library's Document Set content type as the "New Form". Upon creating a New Doc set, you should be routed to CustomDocSet.aspx and realized there were no Save or Cancel buttons!
You can add these buttons adding the following snippet between the & tags:
Afterwards, navigate to the CustomDocSet.aspx page and Edit the page to add a Script Editor webpart for any custom .css and .js
So far so good, as I have not encountered any nuances or issues with this implementation.