Try the following:

    if (document.forms["aspnetForm"]["MSOLayout_InDesignMode"].value == 1) {
       //do your stuff
    }

If the page is in Display mode, you should get an empty string value (""), if it's in Edit mode, it should be "1". This was tested and is working fine for me.