Try the following:
if (document.forms["aspnetForm"]["MSOLayout_InDesignMode"].value == 1) {
//do your stuff
}
OrIf you want to be sure-sure, you can double check with this property:
document.forms["aspnetForm"]["MSOSPWebPartManager_DisplayModeName"].value
which should have a value of "Design"
if a page is in edit mode, or "Browse"
if it's in display mode.
As well you can check in what state the Ribbon is with the following line of code:
SP.Ribbon.PageState.Handlers.isInEditMode();
If the page is in Display mode, you should get an empty string value (""), if it's in Edit mode, it should be "1". This was tested and is working fine for me.