You need to go with People search, instead fetching all users at once. This also fits your requirements for filtering, you can construct different KQL queries for filtering.
Sample code fetching all users using client object model:
var ctx = new ClientContext("https://sp.sharepoint.com/sites/dev");
var keywordQuery = new KeywordQuery(ctx);
keywordQuery.QueryText = "*";
keywordQuery.SourceId = new Guid("B09A7990-05EA-4AF9-81EF-EDFAB16C4E31");
keywordQuery.RowLimit = 100;
var searchExecutor = new SearchExecutor(ctx);
var results = searchExecutor.ExecuteQuery(keywordQuery);
ctx.ExecuteQuery();
Please note, that search results are paged, and for speed and performance it's strongly recommended to implement paging.
For server object model you can use:
using (SPSite siteCollection = new SPSite("<serverRelativeUrl>"))
{
KeywordQuery keywordQuery = new KeywordQuery(siteCollection);
keywordQuery.QueryText = "*";
keywordQuery.SourceId = new Guid("B09A7990-05EA-4AF9-81EF-EDFAB16C4E31");
SearchExecutor searchExecutor = new SearchExecutor();
ResultTableCollection resultTableCollection = searchExecutor.ExecuteQuery(keywordQuery);
resultTableCollection = resultTableCollection.Filter("TableType", KnownTableTypes.RelevantResults);
ResultTable resultTable = resultTableCollection.FirstOrDefault();
DataTable dataTable = resultTable.Table;
}
Additional resources:
Using the SharePoint 2013 search Query APIs
SharePoint Online : Working with People Search and User Profiles
Keyword Query Language (KQL) syntax reference