From experience with several mobile projects: Do not use SharePoint's out of the box mobile master pages. All they contain are simplified list views and you loose everything in your branding, making the site ugly and useless.
Instead, simply turn off the mobile detection for a few devices (smart devices) that are perfectly capable of rendering a normal web page. For publishing sites (low level of complex inputs) this works 100%.
SharePoint 2010 overall works well enough with most of today's smart devices except for some specific bits (most notably the rich text editor which acts a bit quirky), and of course you'll have to zoom around a lot on phones. But I've used the ribbon, etc. on iPads or Android devices.
You can even activate Drag & Drop on your custom interfaces with scripts like JQuery Touch Punch. Note that if you want to go heavy on custom UI, you really, really want to make clear which browsers you will support, and select UI frameworks that already work transparently on all of them.
We've delivered focused drag & drop, responsive portals and parts for SP2010, but never an entire collaboration portal. What is the scope of your project?
As Ronan pointed out authentication can be a pain point because there is no native SSO for most devices. You can get around this somewhat by implementing FBA and "remember me"-like functionality, or using commercial 3rd parties.
If your users will be on 3G (but even on office Wifi), another one of your important challenge will be page load optimization. There are several techniques you can use to reduce that on custom projects (image sprites, minified scripts, putting your production IIS in retail mode, proper use of browser caching, ...)