If you want to hide look at this article. In your situation definition should looks like this one:
<CustomAction
Id="RemoveRibbonButton"
Location="CommandUI.Ribbon">
<CommandUIExtension>
<CommandUIDefinitions>
<CommandUIDefinition Location="Ribbon.DocLibListForm.Edit.Commit.Cancel"/>
</CommandUIDefinitions>
</CommandUIExtension>
</CustomAction>
Small tip: to quickly find button's id you can use inspect html tool in your favorite browser, inspect whole button (it will be html a element with id Ribbon.DocLibListForm.Edit.Commit.Cancel-Large), remove -Large postfix and this is your id in most situations. If button is more complicated, you can find it by this html id in cmdui.xml file, which contains definitions for all ribbon elements.
edit:
If you want display warning, you need replace default button with your own and with your own javascript handler:
<CustomAction
Id="RemoveRibbonButton"
Location="CommandUI.Ribbon">
<CommandUIExtension>
<CommandUIDefinitions>
<CommandUIDefinition Location="Ribbon.DocLibListForm.Edit.Commit.Cancel"/>
</CommandUIDefinitions>
</CommandUIExtension>
</CustomAction>
<CustomAction Id="2C9619A8-CCD8-4984-909C-2C1B7FE794E4"
Location="CommandUI.Ribbon" >
<CommandUIExtension>
<CommandUIDefinitions>
<CommandUIDefinition Location="Ribbon.DocLibListForm.Edit.Commit.Controls._children">
<Button
Id="E6DF7DBA-AC35-4ABA-9A12-4C87DAECA8BE"
Sequence="20"
Command="ConfirmClose"
Image16by16="/_layouts/$Resources:core,Language;/images/formatmap16x16.png" Image16by16Top="0" Image16by16Left="-248"
Image32by32="/_layouts/$Resources:core,Language;/images/formatmap32x32.png" Image32by32Top="-448" Image32by32Left="-288"
LabelText="$Resources:core,cui_ButListFormCancel;"
ToolTipTitle="$Resources:core,cui_ButListFormCancel;"
ToolTipDescription="$Resources:core,cui_STT_ButListFormCancel;"
TemplateAlias="o1"/>
</CommandUIDefinition>
</CommandUIDefinitions>
<CommandUIHandlers>
<CommandUIHandler Command="ConfirmClose"
CommandAction="javascript:if(confirm('Are you sure?')){SP.UI.ModalDialog.get_childDialog().close();}"/>
</CommandUIHandlers>
</CommandUIExtension>
</CustomAction>
Button definition I copy from cmdui.xml, but replace with custom close command.
note: this code replaces all close buttons on all document libraries, if you want limit it only to specific list type or content type, use RegistrationId and RegistrationType attributes on CustomAction.
note2: if you have more complicated confirmation logic, place it in separate javascript file and refer from handler something like this:
CommandAction="javascript:myConfirmationFunction();"