Its all about where are you registering the custom menu, basically if you want to have your custom actions in a list / doc lib, you need to use a List View Command set, where you need to set the proper location, so where the commands are displayed. The possible values are:
- ClientSideExtension.ListViewCommandSet.ContextMenu: The context menu of the
item(s).
- ClientSideExtension.ListViewCommandSet.CommandBar: The top command set menu in a list or library.
- ClientSideExtension.ListViewCommandSet: Both the context menu and the command bar (corresponds to SPUserCustomAction.Location="CommandUI.Ribbon").
Resulting in having your commands in the list menu like shown here:

Take a look at the following article: https://docs.microsoft.com/en-us/sharepoint/dev/spfx/extensions/get-started/building-simple-cmdset-with-dialog-api#debugging-your-listview-command-set-using-gulp-serve-and-query-string-parameters