There are several ways to achieve this.
I think the most preferrable way is to create a computed field.
1. Computed field
The point is, actually there are 5 "Title" fields in any SharePoint list:
First stores the data, the other 4 are Computed and simply generate some HTML, using data from the first column.
You can use similar approach and add your own computed field into the list, and then add it to the list view.
To add a computed field, you can use AddFieldAsXml method.
Example of using AddFieldAsXml for Computed field creation:
P.S. Most safe way to address edit form is to use ListForm.aspx page, as follows:
/_layouts/listform.aspx?PageType=6&ListId={PUT-LIST-GUID-HERE}&ID=5
(PageType value goes from PAGETYPE enumeration)
2. XSL Transformation
Another option is to use XSLT capabilities of the XsltListViewWebPart. In this case, you need to write your xsl file, and attach it to the list view, for example by filling in SPView.XslLink property.
Example:
<xsl:stylesheet xmlns:x="http://www.w3.org/2001/XMLSchema" xmlns:d="http://schemas.microsoft.com/sharepoint/dsp" version="1.0" exclude-result-prefixes="xsl msxsl ddwrt" xmlns:ddwrt="http://schemas.microsoft.com/WebParts/v2/DataView/runtime" xmlns:asp="http://schemas.microsoft.com/ASPNET/20" xmlns:__designer="http://schemas.microsoft.com/WebParts/v2/DataView/designer" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:SharePoint="Microsoft.SharePoint.WebControls" xmlns:ddwrt2="urn:frontpage:internal" xmlns:o="urn:schemas-microsoft-com:office:office">
<xsl:include href="/_layouts/xsl/main.xsl"/>
<xsl:include href="/_layouts/xsl/internal.xsl"/>
<xsl:param name="AllRows" select="/dsQueryResponse/Rows/Row[$EntityName = '' or (position() >= $FirstRow and position() <= $LastRow)]"/>
<xsl:param name="dvt_apos">'</xsl:param>
<xsl:template match="FieldRef[@Name='LinkTitle']" mode="Computed_LinkTitle_body">
<xsl:param name="thisNode" select="." />
<a>
<xsl:attribute name="href">/_layouts/listform.aspx?PageType=6&ListId=<xsl:value-of select="$List" />&ID=<xsl:value-of select="$thisNode/@ID" /></xsl:attribute>
<xsl:attribute name="onclick">
SP.UI.ModalDialog.showModalDialog({
url: '/_layouts/listform.aspx?PageType=6&ListId=<xsl:value-of select="$List" />&ID=<xsl:value-of select="$thisNode/@ID" />'
}); return false;
</xsl:attribute>
<xsl:value-of select="$thisNode/@Title"/>
</a>
</xsl:template>
</xsl:stylesheet>
P.S. This code has been tested and works.