in SharePoint 2013 can we use HTML 5.0 and css 3.0 on non-publishing site e.g Team site? In this case publishing feature is not activated in the team site. What is best practice? Or should we use HTML 5.0 and CSS 3.0 only for publishing site?
Best practice is to activate the "SharePoint Server Infrastructure Publishing feature" on the site collection level and activate the "SharePoint Server Publishing feature" on the team site. Make a copy of the master page you want to use as template, give it a unique name and start branding. Never ever edit the default files of SharePoint since you may break something, or future updates of the file may erase your custom branding.
You can use HTML5 and CSS3 on a real publishing site or a team site with publishing features activated. It's possible to do branding without publishing features, but it's not recommended.