There is a good post that can resolve your problem.
Step 1: Create a document library. I’m naming it Restricted views. It is better to not show this library in quick launch because it’ll be used to host the all tasks view.
Step 2: Now open SharePoint Designer 2010, go to All Files – Lists – Tasks and copy AllItems.aspx (This is All Items view)
Step 3: Go to All Files – Restricted Views (It is the document library which we created in step 1) and paste the AllItems.aspx there
Step 4: Go to All Files – Lists – Tasks and delete AllItems.aspx
Step 5: Open the SharePoint 2010 portal – Site Actions – View All Site Content – Restricted Views, you will see AllItems.aspx there.
Step 6: Select AllItems.aspx – Document Permissions – Stop Inheriting Permissions and then grant permissions to manager.
Step 7: Go back to SharePoint Designer 2010, go to All Files – Lists – Tasks and then delete AllItems.aspx
Step 8: Go to All Files – Restricted Views, copy AllItems.aspx and paste it to All Files – Lists – Tasks.
Step 9: Open the SharePoint 2010 portal, go to Task List, when you click the views drop down, you’ll see two views titled All Items, don’t worry.
Step 10: Go to list settings – views and click All Tasks views, now you’ve to be careful here and delete the old view
Make sure URL of the view says /Lists/Tasks/AllItems.aspx and not /Restricted Views/AllItems.aspx, this is important because the view in Restricted Views library is the one with proper permissions.
Step 11: Now trying accessing the view as manager, you’ll be to see it and when you’ll try to access the view using any user who you didn’t give, they’ll see access is denied when they’ll try to access it.