We use custom MSBuild tasks with Visual Studio to monitor changes and then robocopy changed files to the 12 hive. We also monitor *.cs files and deploy the assembly to the GAC when code changes. This can be done relatively easily using the Inputs/Outputs attributes of an MSBuild target.
<ItemGroup>
<inputFiles Include="$(ProjectDir)12\TEMPLATE\FEATURES\MyFeature\*.*" />
</ItemGroup>
<Target Name="DeployLocalFiles" Inputs="@(inputFiles)" Outputs="$(ProjectDir)obj\$(MSBuildPRjectName).tmp" >
<Exec Command="robocopy.exe "$(ProjectDir)12\TEMPLATE\FEATURES\MyFeature" "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\TEMPLATE\FEATURES\MyFeature" /E /XA:H /XD .svn" IgnoreExitCode="true" />
<Touch Files="$(ProjectDir)obj\$(MSBuildPRjectName).tmp" AlwaysCreate="true" />
</Target>
By also disabling output caching on the local environment we can work fairly painlessly and see our changes immediately by hitting F6.
This doesn't solve the underlying issue of deploying to a test server, but does allow much easier and faster testing locally which can speed up the development process.
For deployment to other test environments we have powershell scripts that automate the stsadm commands and use Gary Lapointes excellent gl-execadmsvcjobs extension to ensure commands are flushed across all servers on the farm when executed in sequence.