This was tested on a non-branded, out of the box enterprise wiki page. The behavior is consistent.
Case 1 (as expected): I have an unmodified document library 'Documents' and a pages library 'Pages' with wiki-pages. When I edit a wiki page and in the ribbon choose 'Upload a document', I get the dialog for uploading and choose to upload to the Documents-library. I'm then back on the wiki-page where an document type icon and a link to the uploaded document have been inserted in the text.
Case 2: Same as case 1, but I now have added a site column, e.g. 'URL', to the document library. When I choose to upload a file to the document library I then get another form asking me to fill out the value of the 'URL'-column. After saving the form, I'm returned to the wiki-page, but no link to the uploaded document have been added to the text.
With the browser console open, I see that I get a javascript-error in the file sp.ui.rte.js "Uncaught TypeError: Cannot read property 'newFileUrl' of undefined" after clicking save. As this file is minimized it's a bit hard to specify exactly where it is, but it is in the function $FG_0.
var b=document.createElement("A");
b.href=a.newFileUrl;
Workaround for now is to remove the extra columns of the document library, but that is unfortunate. Any ideas if it's possible to fix?