There is an identified issue with SharePoint 2013 ignoring the values set through the
See: Custom Access Denied Page in SharePoint 2013.
Although the same UpdateMappedPage method and Set-SPCustomLayoutsPage cmdlet exists in 2013, there is an identified bug in the product related to the property. The custom location can be set using either of these methods, but SharePoint will not recognize them and will continue to use the out of the box accessdenied.aspx page.
The comments on that page suggest that it is expected to be patched Feb. 2014.
Additionally, in SharePoint 2013, the path to files in the layouts folder has changed. If the files are in the 15 hive, your path should be
The article Virtual directories in SharePoint 2013 solutions explains this as:
For files in the virtual folders _layouts or _controltemplates, SharePoint needs to use the version of the files in %ProgramFiles%\Common Files\Microsoft Shared\web server extensions\15\ (sometimes called the 15 hive) or in the corresponding 14 hive, depending on the mode of the site collection. SharePoint adds “/15” into the virtual directory path just after the virtual directory name to signal that the SharePoint 2013 files should be used. The absence of that extra string indicates that SharePoint 2010 files should be used.