This is something I've been looking into for a while.
We use sharepoint here as well as our own bespoke bit of software that lets you create and edit our own type of documents.
I've written a COM dll, which I've tested with the following code as a vbs script:
Dim obj
set obj = CreateObject("Ctest.SharePoint.OpenDocuments.1")
msgbox obj.EditDocument("c:\mytest.olf", "06FC39A0-2CAA-11D0-B6B7-004095131445")
That works a treat.
I've added our file extension to Sharepoint search/indexing.
If I edit DocIcon.xml with the line:
<Mapping Key="olf" Value="olf.png" EditText="My Application" OpenControl="Ctest.SharePoint.OpenDocuments.1"/>
I get the "The document could not be opened for editing. A Microsoft SharePoint Foundation compatible application could not be found to edit the document." error.
Any idea's on this as I'm not sure what else I can do!