I'm trying to customize my SharePoint site to be an internet facing site.
I know I can do this by setting up customized masterpages and page layouts, but I didn't find a lot of documentation for this.
What do I need to do to setup my site?
The MSDN featured an article describing a real world scenario for branding SharePoint 2010 for an internet facing website: http://msdn.microsoft.com/en-us/library/gg430141.aspx
The above link is valid, but i would say that is more than that, if you want a proper experience. You need to consider URL addressing, Authentication mechanisms (Claims, Windows), user store, anonymous or not, whether you are extending one SP Web App or just starting with a dedicated one, what services would you offer, would you enable RSS, would you use CQWP for its rollup capabilities, etc.
Actually Waldek Mastyrkasy has a series of improving the actual experience via a study case, start here http://blog.mastykarz.nl/how-we-did-it-mavention-nl-part-1-branding/