I don't think it would be possible apply spfx extension on name field(certain other columns also) but it does work on FileLeafRef column. Ref link
Column formatting could have been an option but problem is that Name column does on on appear column settings, it just appears in view fields with this 3 columns
Name (linked to document).
Name (for use in forms)
Name (linked to document with edit menu)
Another idea might be to use a calculated field based on name column and using column formatting. But then again search results would again need to be customized using search display template.
Unfortunately, I don't think there would be any work around, because Name column just stores document name and url to document is generated dynamically in all OOTB views(classic and modern view) based on column selected in view. Same case goes with search result, it uses FileLeafRef to generate link to document.