As per title. I want to grant only certain permission groups/users the rights to removing a web part from a page. Is this possible?
Yes it is possible, only users with Full Control, Design or Contribute permission can only add or remove webparts as demonstrated in picture below,
Don't forget about custom permissions levels.