Ahh, now I see what you're doing. While your approach will work, I tend to prefer the "use a supported method for everyone" approach. The issue with doing this client side is that even if you remove the Silverlight container with Javascript, the browser is still initially going to try and load it.
This is one of my gripes with the SharePoint team; while I understand that Silverlight is a Microsoft product, they've put so much work into making SharePoint 2010 cross browser compatible, but they're still grasping onto the straw that is Silverlight. A nice cross browser compatible video web part would have been far more useful.
All that said, in my opinion, you're better off ditching the Silverlight component, and using something like VideoJS in a Content Editor Web Part. It's certainly a bit more "work" to get stood up than just dragging in a web part, but it handles everything cross browser, unlike the OOTB Silverlight player.