It is important to understand that the Notes are considered "Social Data" or content and are not stored within the site. The social data is stored centrally as part of the associated User Profile Service App and related databases, and only reference your site, page, or content. When you put the Note Board Web part on the front page of a team site, it will show any notes for that page, but it is unique to the page and not the site as a whole.
If the page url were to change for example, the note board would actually appear to not have any notes until new ones are added.
With all of that in mind, since the notes are not site content but rather social data, the site's security trimming does not come into consideration and it would not be possible to prevent it from going through the activity feed without some sort of customization that filters the activity feed listener.