I am troubleshooting a poorly performing SharePoint Search Crawl. Background? 2 Million items, Single Crawl Server crawling itself with 4CPUs and 16GB of RAM. It uses 100% RAM for a while and hits around 40 items per second. However, it slows way down to 5 - 6 items per second and does not use the RAM/CPU all the way anymore. SQL Cluster back-end is barely being touched, so I don't think it is SQL. Any idea why the crawl is taking around 24 hours?
Anytime you hit 100% RAM at all, overall system performance tanks to a fraction of normal as it is now constantly swapping to disk for almost every task. The more memory intensive the underlying process is, the worse the problem becomes.
In addition to what @Cory says, you should also make sure that you aren't doing other processes at the same time, like SharePoint/AvePoint backups. If SharePoint search is not your Enterprise search but its data is included in the Enterprise Search, then you might want to find out when that crawler is hitting your system as well.
There are about a dozen other tweaks that you could do but the end result may well be that you have exceeded the limitations of a single server and that it is time to throw more hardware at the problem so that you can split the load.
Is this for SharePoint 2007 or 2010? It sounds very similar to the farm I am working with. I have a little over 2 million items as well with one index server and 2 query servers. Except that a full crawl takes 40-60 hours. Are you talking about a full crawl or an incremental? Check the disk space on both your query and index servers and make sure there is plenty of room. If the servers are low on space propagation may pause until space is freed up this will make your crawls last longer. If it is a full crawl I would take a look at your needs, maybe just have fulls run on the weekend. My incrementals finish in 2 to 7 hours. In all reality you should probably add another index server, I wish I could.