Since every item in SharePoint (e.g. Document Library's, Lists, Surveys, Blog Posts/Comments, etc.) allows for modular security that can be very granular and is different from the parent site it lives within.
I've done the same thing many times with all the different components of SharePoint. What I would do is on the site that holds the survey first make sure that small group already has Visitor level, creating a new SharePoint Security Group called something like 'Survey Members' and putting them in there either by an Active Directory group or by individual names. Then split the permissions off from the parent site which copies them (Survey Settings-->Permissions menu) along with your new Security group. In the same Permissions menu click on the Survey Members group name to change its permission level. On the new screen click the Members check box and uncheck the Visitors box.
Now those users will have the ability take and/or edit surveys depending on the other Survey settings.
Hope that helps.