Items displayed in a Content Search web part is controlled by the chosen display template in the web part properties.
To make the results open in a modal dialog, go to the site collection where the web part is > site settings > Master pages and page layouts > Display templates > Content Web Parts.
Download a copy of the display template HTML file that you are using to display your data, for example Item_TwoLines.html
Open the display template in any text editor ( notepad or notepad++ ).
Begin with changing line 3 so that you know which one is your custom display template, for example
<title>Two lines Custom</title>
Then replace the following value on line 67 ( Item-TwoLines.html. )
<a class="cbs-Line1Link ms-noWrap ms-displayBlock" href="_#= linkURL =#_" title="_#= $htmlEncode(line1.defaultValueRenderer(line1)) =#_" id="_#= line1LinkId =#_">_#= line1 =#_</a>
Go back to the site where your content search web part is and edit the web part, you should now be able to set your custom template "Two lines Custom" as the Display Template to use. Select it and save the page. When clicking on a result, it should now open in a dialog.