I normally program on my host's copy of Visual Studio unless the client explicitly requires the use of the Built in Visual Studio SharePoint projects. (SharePoint is not installed on the host)
You need to create a backup of the SharePoint assemblies found in the 12/14 hive on your host most can be found in the ISAPI folder. You can then reference the assemblies with in Visual Studio from this local location. Visual Studio includes Assembly Location hints that are sent to the compiler.
It is also worth while coping the XML schema so you can get intellisense with in xml files, but you will have to add the schema to your Visual Studio.
As you are unable to use the "Magic" Visual SharePoint project types you will ether need to have a hand crafted WSP build script using MakeCab and the relevant Manafest.xml file or use one of the WSP communality solutions (E.G. WSP builder ect. though many are not being maintained any more).
It becomes a bit more complicated for Remote Debugging, there are issues with cross domain debugging so this can take some time to configure. You need to store the PDB files on the target, but it is all quite possible. Note there are security issues with Remote Debugging across domains so be sure to only enable this on your Host to VM network only as you user name and password are transmitted unencrypted.
Alternatively you could just access your source control and do any debugging on the VM's copy of Visual Studio.
I would recommend build scripts in powershell that copy the WSP and deploy it as well as copying the relevant PDB files.
Have a look at Chris O'Brian's Continuous Integration posts to help with the automation of moving the Wsp from the Host to the VM.
I have used this approach for 5 years and have had no issues.