I don't think you can do it through the 'Custom Action' function. You'd probably have to build a custom action using Visual Studio.
Another option - the one I used - is to replace the list view with a dataview web part built in SP Designer. Then at the top of your DVWP you can add a toolbar with whatever buttons you want. Using JavaScript, you can grab the RootFolder parameter from the URL and then pass it to your button like so:
<input type='button' value='Explore' onClick='javascript:CoreInvoke("NavigateHttpFolder","" + RootFolder + "","_blank");'/>
In 2013, you can do the same thing using a JSLink file instead of needing a DVWP, which IMO is a lot easier. JSLink is one of the better features offered by 2013 over previous versions.
Note that you'll need to tweak that button code for it to work, but the important part is that you call the CoreInvoke function and then pass the 'NavigateHttpFolder' operation and the URL value. Also, keep in mind that the RootFolder parameter is relative, so you may have to build out the first part of the URL in your script prior to passing it.