Unfortunately, there's nothing in the view initially that will allow you to filter this information. I'd suggest creating a Manager field on the task list, and then developing a simple SharePoint Designer Workflow that looks up the manager for you, and inserts that value into your newly created Manager field. After you do this, you can create filters off that value, including a Manager View where "Manager=[Me]." Note this isn't changing who has access to what records -it's just facilitating the view so that data can be filtered to a specific manager.
How you grab the manager in your workflow depends on whether you're creating a SharePoint 2010 or 2013 Workflow. This blog does a good job covering how you accomplish this in both scenarios.