Yes, if it is not properly removed from the farm then you will be in trouble regarding to backup and restore.
When you install SSRS add-in, it will deploy some features in SharePoint, Features include Power View, a Report Viewer Web Part, a URL proxy endpoint, Reporting Services content types and application pages so that you can create, view, and manage reports, report models, data sources and other report server content on a SharePoint site.
MSFT mentioned two method to remove it. You have to try one of the method and then delete the left over manually i.e permission groups, files uploaded to sharepoint, database created using SSRS etc.
- Using control panel will run a complete uninstall of the files on
the current computer AND it will remove the Reporting Services
object and features from the SharePoint farm. When the Reporting
Services object and features are removed you can no longer review
and update reports.
- The command line method to uninstall the add-in allows you to use
the LocalOnly parameter to only remove the add-in files from the
local computer and the Reporting Services object and features in the
farm will not be changed.
Uninstalling the add-in will remove server integration features that are used to process reports on a report server. It will also remove the Reporting Services pages from SharePoint Central Administration and other custom Reporting Services pages. You may also want to remove any reports and other report server items that you no longer use on the affected SharePoint sites. They will not run after the Reporting Services Add-in is removed.
How to Remove the Reporting Services Add-in