yes you can very well achieve this functionality. All you have to do is create a list which will contains the required user's emails. Then you will have a data connection to this list.
You can add a hidden text box inside infopath form and with help of getuserprofilebyname web service you can get current user's email id. The hidden text box will have the user email id only if user belongs to that list.
Then you can write condtional rule on display properties of text field-
rule- if HiddenTextBox value is blank, hide this textbox.
Conclusion if user belongs to this list then he can see the field otherwise he will not.
Let me know if u need further inputs.