The Search Result web part has a set of pre-loaded display templates. So that is the reason you see templates get loaded that aren't necessary.
The list of pre-loaded templates is the following:
"~sitecollection/_catalogs/masterpage/Display Templates/Search/Group_Default.js",
"~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_Default.js",
"~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_Site.js",
"~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_Word.js",
"~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_PowerPoint.js",
"~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_Person_CompactHorizontal.js",
"~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_BestBet.js",
"~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_WebPage.js"
These templates can be defined with a web part property called: PreloadedItemTemplateIds. This property is not available in the UI, so you need to export the web part and update the PreloadedItemTemplateIds element and upload it back to the site.
Before:
<property name="PreloadedItemTemplateIdsJson" type="string">["~sitecollection/_catalogs/masterpage/Display Templates/Search/Group_Default.js","~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_Default.js","~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_Site.js","~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_Word.js","~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_PowerPoint.js","~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_Person_CompactHorizontal.js","~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_BestBet.js","~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_WebPage.js"]</property>
After:
<property name="PreloadedItemTemplateIdsJson" type="string">[]</property>
Important: you need to leave the brackets, if you remove them, the web part will load a hard coded list of display templates (which is just the same as the default property setting).