I'm not a Silverlight expert, but it should be entirely possible to do what you want. The Client Object Model presents not only lists and list items, but also webs too - since these are what the navigation links are too, all the data you need can be obtained this way.
Effectively you would need to use the following approach:
- Get a reference to the current site collection
- Iterate/find each of the top-level webs you wish to use in navigation
- For each, find the child webs according to how many levels deep you wish to go
- Do the Silverlight bit to create menus from this data
Remember that SharePoint's navigation is quite optimized (caching etc.), and is also fairly configurable (e.g. how many levels deep, starting node, add authored links etc.). In a completely custom implementation, you'll need to establish how far you need to go. Code following the above pattern would be quite inefficient if written server-side with the SharePoint .Net API, but I think you might be OK because the Client OM will batch up the requests to the back-end.