I've looked through some books and read a bit online, but am unclear on a particular point. I have a project where we want to use a completely custom layout on top of a SharePoint Foundation 2010 site. Is this even feasible?
You can create a theme very easily by using Microsoft Theme Builder. But to do a complete look and feel you are going to have to look at time vs effort vs effectiveness. Branding SP 2010 is very feasible.
If your site is a company Intranet, try using a custom theme and then adding a picture banner, this is minimal work and can make a site look very nice. If you have an external facing site (company website) I would suggest a full branding planning session to see how you want to move ahead. You can take a look at http://sp365.co.uk/tag/branding/ for a a great step by step tutorial on how to brand SP2010
It is completely possible to do on top of SharePoint Foundation - it supports all the normal themes and master pages and so one. It has it's limitations, being free that shouldn't be a surprise, for example if you going to have multiple sites and want to share common master pages it is a pain, where this is simplified in Standard and above.
If you going for a website (which would likely be one site anyway), I see no reason why not to use Foundation.
Feasibility is a relative term, really depends how much time+effort you're willing to put in.
But yes, it is completely possible.
A tutorial to get you started: http://blog.henryong.com/2010/06/08/how-to-create-custom-sharepoint-2010-page-layouts-using-sharepoint-designer-2010/
I found this article which helps out with EXACTLY what I needed to do in this case. Adding link for anyone else who comes across this. http://msdn.microsoft.com/en-us/library/gg430141.aspx