In fact to manage a SharePoint site's navigation, one must have either Full Control or Design level permissions for the site (source). As both of these are high-level permissions, it would likely lead to extreme permission level policy where many of the contents would have unique permissions. Even if that would be well managed, it would still be poorly administrable.
I can only come with a workaround, where you'd define your managed as Managed Navigation based on a term set. With term sets you can define term set contributors who have the permissions to manage terms, which in your case could be navigation's links. The drawback with this is the manually updated navigation and another permission group to take into account. For more information, see Assign roles and permissions to manage term sets.
I understand there are scenarios where your requirement is very much requested, but allow me to throw my 2¢. If I wouldn't be able to trust my users to correctly handle lists (not to mention any "bigger" SharePoint's content areas) - which is a relatively normal case - , I wouldn't let them configure the navigation either.