I think you cannot. Everything turns around the user.
The only way now I can figure out is to fetch each user profile and using SPSocialActor class getting if site is followed by that user. If the site is followed you can add the user to a list of users (List) with properties as imgUrl, name, surname, etc.
Try this (I have not tested it) but I am not sure if this will work but you can follow this approach
SPSite currentSite = SPContext.Current.Site;
SPServiceContext contexto = SPServiceContext.GetContext(currentSite);
UserProfileManager perfiles = new UserProfileManager(contexto);
foreach (UserProfile userProfile in perfiles)
{
try
{
//Check if the user profile is already created
if (userProfile.PersonalSiteInstantiationState == PersonalSiteInstantiationState.Created)
{
SPSocialFollowingManager followingManager = new SPSocialFollowingManager(userProfile, contexto);
//Get the sites followed by the user
SPSocialActor[] followedUserSites = followingManager.GetFollowed(SPSocialActorTypes.Sites);
//now get if the site is in followedUserSites array
SPSocialActor matchSite = followedUserSites.Where(st => st.Uri.AbsoluteUri.Contains(currentSite.Url)).SingleOrDefault();
if (matchSite != null)
{
//Add logic to your custom object or whatever
}
}
}
catch(Exception ex)
{
}
}
Hope it helps