You can associate your custom action to a specific list by using the RegistrationId
property.
Example:
<CustomAction ControlAssembly="MagicalProject.Actions"
ControlClass="MagicalProject.Actions"
ControlSrc="/_layouts/MagicalSite/MagicalPage.aspx"
Description="This is my custom action which does magical stuff"
GroupId="MyMagicalActions"
Id="f5bcd75e-23b9-4623-9a82-41cc7ceb10a6"
RegistrationId="MyMagicalList" // Or whatever other identifier you have for your lists
RegistrationType="List"
RequireSiteAdministrator="false"
Rights="AddListItems"
Sequence="1"
ShowInReadOnlyContentTypes="false"
ShowInSealedContentTypes="false"
Title="MyCustomAction">
UPDATE: It also applies for ContentTypes. So if you associate it with a contenttype, then all lists inheriting that specific contenttype will also get this custom action. In case you wanted to know how to put it on more than 1 list.
All it takes is changing RegistrationType
from List
to ContentType