I think most of the functions you need are in Form.js in the 12\TEMPLATE\LAYOUTS\1033 folder (assuming an english install of WSS 3.0/MOSS 2007.) You may also need HtmlEditor.js and AssetPicker.js depending on which aspects of the toolbar you are rendering. I have never even attempted to do what you are suggesting and these functions are not documented so you will need to work that out for yourself.
Is it seriously a business requirement that the whole list form renders in a dialog? Or are you just establishing a POC/R&D?
Additionally you could use the Telerik RADEditor Lite for MOSS its the last link on the page, oh yes and its free. Its also cross browser compatible (if that is an authoring issue for you.)
What about using TinyMCE? http://tinymce.moxiecode.com? That one's very easy to implement.