The LINQ tools are very handy for developers, but they were never guaranteed to perform better than ugly CAML code or be more resilient to user changes (hint: they rely on display names that may change), for that matter.
Look into ULS logs (
Set-SPLogLevel -Trace Verbose) for answers in the Database category.
Moreover, I assume you're doing this from a Console executable on a development machine, right?
Try SQL Server Management Studio's profiler or other tools to understand what's the load on SQL Database, how are the indexes/indices and execution plans (mis)used. Sometimes you might find useful to check for database fragmentation.