I want to create a custom edit form for a picture library, and change the default edit form for all picture libraries to a custom edit form.
My question: how to create such custom edit form for a picture library?
You might want to take a look at creating a custom list definition that uses the existing picture library as a starting point. (12\Template\Features\PictureLibrary)
As part of that process, you could specify your custom editform.aspx
Once you have a list definition, all new picture libraries created from your new template will use the custom edit form.
However if you need to change the edit form for ALL existing picture libraries, you will probably need to edit the file in the 12 hive, which is NOT recommended
C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\FEATURES\PictureLibrary\PicLib\EditForm.aspx