The short story:
Usually you'll set the master page as both default and custom master page, the most common exception being a very customized master page for publishing which can't be used for ordinary pages in which case you'll only set it as custom master page.
The full story:
When creating pages in SharePoint your main options for setting the MasterPageFile property are one of two tokens "~masterurl/default.master"
or "~masterurl/custom.master"
the latter is mainly used for publishing pages as you can read from the Site Master Page Settings page:
Site Master Page = ~masterurl/custom.master = Custom master page
The site master page will be used by all publishing pages...
System Master Page = ~masterurl/default.master = Default master page
Use the system master page for all forms and view pages in this site...