You can set a custom active directory query. This is done at the web application level. Here is an example using PowerShell:
$webapp = Get-SPWebApplication http://yourwebapp.com
$webapp.PeoplePickerSettings #Displays your current values
#Finds users with an email address, who are not disabled, and account name
#not ending in VDI or group name not ending in -Test
$webapp.PeoplePickerSettings.ActiveDirectoryCustomFilter = "(|(&((objectCategory=Person)(objectClass=User)(mail=*)(!userAccountControl:1.2.840.1135126.96.36.1993:=2)(!sAMAccountName=*-VDI)))(&((objectCategory=Group)(groupType:1.2.840.1135188.8.131.523:=4)(!cn=*-Test))))"
$webapp.Update() #commits the change
$webapp.PeoplePickerSettings #Displays the updated values
This query will trim users with account names ending in -VDI and groups with names ending in -Test. Here is a breakdown of the query with comments (making it more of a psuedo query, since LDAP queries can't actually have comments):
(| // or a person or group
(& // a person must match all of these conditions
(mail=*) // mail is not empty
(!userAccountControl:1.2.840.1135184.108.40.2063:=2) // account not disabled
(!sAMAccountName=*-VDI) // account name doesn't end with -VDI
(& // a group must match all of these
(groupType:1.2.840.1135220.127.116.113:=4) // domain local group
(!cn=*-Test) // name doesn't end with -Test
Note: I don't have an easy way to test this at the moment, so my syntax may not be perfect but hopefully you get the idea. You'll need to adjust this to your needs, for instance you may not want only domain local groups.
Ref: Another People Picker Post – SharePoint 2013 and SharePoint 2016
Should work the same for 2013.