Thats a very high level list of requirements so I can only give you some pointers :-
Free only if you're using internally and already have Client CALS (How Much does SharePoint Cost)
SQL Server Express 2008 R2 has a 10GB limit - beyond that you have to look at remote blob storage or a full version of SQL Server (cost implications)
Document workflow - built in with 3rd party tools such as K2 Black Pearl, Nintex and others (no affiliation)
Notification - built in Alert Me but somewhat limited (e.g. hard to customise, can't do date based alerts etc) but there are lots of other ways you can do this (workflow/event receivers/open source and 3rd party tools)
Filters - static filters via web part views, filter web parts built into Enterprise version and plenty of 3rd party filter web parts available if you're using Foundation.