When do you create a custom page layout vs. a custom master page for a site? Is it good practice to create custom page layouts only for limited branding of SharePoint 2010? I'm thinking the page layout is the actual body of the page, while the master page is the wrapper. If that's the case, then does the page layout provide finer control over the editable area of the page than the master page? Your clarification will be appreciated. Thanks.
In the most concise terms I can think of - a Page Layout is used to display content, a Master Page governs the look and feel for the whole site.
A Page Layout has an associated content type, therefore it stores content, and a page derived from a Page Layout will store the content in the content type's fields.
A MasterPage does not have a content type and thus is not designed for storing content.
You can have branding applied to an entire site through the masterpage, but if you want a specific page in the site to have something looking slightly different, you can load a specific CSS file in that Page Layout, and all pages deriving from that Page Layout will load that CSS.
So, to answer your questions:
When do you create a custom page layout vs. a custom master page for a site?
Depends what you want to do with branding. Master Pages are global, Page Layouts are for a specific "type" of page/content.
Is it good practice to create custom page layouts only for limited branding of SharePoint 2010?
If a style of page will have a unique look and feel that will remain consistent with that page, but different from the rest of the site, this is an acceptable method (ie: loading a custom CSS within that page layout).
If that's the case, then does the page layout provide finer control over the editable area of the page than the master page?
You can think of the master page as the web-site "chrome". This will have markup and controls that will be consistent across your whole site. You can have different pages in a site associated with different master pages but this is unusual.
The page layouts (and this only applies to publishing sites) are used to control the structure of a category of pages in the site, for example an article page. Usually the elements coming from the page layout will be "inside" the elements of the master page but the latter can be overridden. The actual content will come from fields from items in the "Pages" library.
In terms of how it actually works, you should think of the page layout as being equivalent to the aspx page of a non-publishing site, rather than the "Pages" in your Pages library which only contain raw content.
I was asked this question in an interview. The answer is clearly described in the below link.