I am trying to open a video within the webpage in Sharepoint 2013 asset library, but it always open the file instead of play on the webpage. can you please help?
It sounds like the Content Type is wrong, is it associated with video content type? If so, it should direct you to the video player page and you can play your video from there.
Default SharePoint video player in Asset library supports only several video formats but not all. For example, clicking a .flv file in an Asset library results in opening the display form instead of the player. To distinguish between supported or unsupported videos, you can check the video icon in the default SP view in asset libraries.
This is how a supported video looks like:
This is how an unsupported video looks like:
To resolve this, I would convert unsupported videos to MP4 using video conversion programs before uploading to SharePoint.