You could download the AppFabric installer manually (but don't run it) http://go.microsoft.com/fwlink/?LinkId=235496
, and then run the installer through the Prerequisites installer like this: prerequisiteinstaller /appFabric: WindowsServerAppFabricSetup_x64.msi.
There is more info at this link: http://blogs.technet.com/b/meamcs/archive/2012/12/26/install-and-download-sharepoint-2013-prerequisites-offline-manually.aspx
In my experience, if the AppFabric service isn't installed 100% correctly, you will have all kinds of sporadic SharePoint problems (crashes) in the future. I also suspect (but have never confirmed) that trying to run the prerequisites installer multiple times sometimes gets the machine into a state where the App Fabric service can never be installed, so you might consider re-installing the OS if this keeps happening.
I've also found that the AppFabric service installation has problems if the machine (and the user account you are installing under) are not correctly joined to the AD domain. You might check the clock on your VM to make sure its within a few minutes of the clocks on your domain controllers, and also make sure that your new VM is only using Domain Controllers as its DNS servers (so it can look up all those SRV records, which other DNS servers may not have).