I don't think you can do it through the 'Custom Action' function. You'd probably have to build a custom action using Visual Studio.
In 2013, you can do the same thing using a JSLink file instead of needing a DVWP, which IMO is a lot easier. JSLink is one of the better features offered by 2013 over previous versions.
Note that you'll need to tweak that button code for it to work, but the important part is that you call the CoreInvoke function and then pass the 'NavigateHttpFolder' operation and the URL value.
Also, keep in mind that the RootFolder parameter is relative, so you may have to build out the first part of the URL in your script prior to passing it.