I've looked through some books and read a bit online, but am unclear on a particular point. I have a project where we want to use a completely custom layout on top of a SharePoint Foundation 2010 site. Is this even feasible?
You can create a theme very easily by using Microsoft Theme Builder. But to do a complete look and feel you are going to have to look at time vs effort vs effectiveness. Branding SP 2010 is very feasible.
If your site is a company Intranet, try using a custom theme and then adding a picture banner, this is minimal work and can make a site look very nice. If you have an external facing site (company website) I would suggest a full branding planning session to see how you want to move ahead. You can take a look at http://sp365.co.uk/tag/branding/ for a a great step by step tutorial on how to brand SP2010
It is completely possible to do on top of SharePoint Foundation - it supports all the normal themes and master pages and so one. It has it's limitations, being free that shouldn't be a surprise, for example if you going to have multiple sites and want to share common master pages it is a pain, where this is simplified in Standard and above.
If you going for a website (which would likely be one site anyway), I see no reason why not to use Foundation.
Feasibility is a relative term, really depends how much time+effort you're willing to put in.
But yes, it is completely possible.
A tutorial to get you started: http://blog.henryong.com/2010/06/08/how-to-create-custom-sharepoint-2010-page-layouts-using-sharepoint-designer-2010/
If you want to customize the layout of the SharePoint Pages, try to customize the master page and Page Layouts. By Using Theme, we can only customize the colors of the SharePoint pages.
I found this article which helps out with EXACTLY what I needed to do in this case. Adding link for anyone else who comes across this. http://msdn.microsoft.com/en-us/library/gg430141.aspx