According to the documentation on MSDN, you should use the following:
source: An ASP.NET resource expression in the form $Resources:keyname, where keyname is the name half of a name/value pair in a resource file (.resx).
defaultResourceFile: The base file name of the language resource file containing a localized string value. For example, if you have a series of resource files named myresources.en-us.resx, myresources.es-es.resx, myresources.de-de.resx, and so on, the value to pass in this parameter is myresources.
In your case, I would think you should use:
GetLocalizedString("$Resources:custList", "core", 1033) //1033 for English
I got the values from looking at the CustomList definition at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\FEATURES\CustomList\ListTemplates\CustomList.xml on my SharePoint installation.