We have an intranet farm with some farm solutions deployed. They fetch data and display it to users from Oracle database. We have 4 web front end servers.
We are facing slow performance issue. CPU usage of WFE servers is very high and goes upto 100% even though each server has a RAM of 16 GB.
Please suggest which tools can I use to track down the actual issue and the culprit causing it?
Check your logs, application and ULS. Any errors or anomalous behaviour?
Have you checked the DB? Most bottlenecks occur there. Speak to your DBA he should be able to determine any issues.
Use Perfmon to establish a detailed picture of what is happening. There are tons of resources for this just Google it. Once your problem is resolved be proactive and baseline your performance and then check for variance regularly.
Are all 4 servers flat-lining at 100%? Note that spiking to 100% is fine as long as it is infrequent spikes. What is the average CPU usage? High != bad as you don't want your servers to be a wasted resource. Is IIS healthy? Check for app pool recycles.
What about the solutions you are using is there a higher usage level? Were they load tested? Do they write out to the ULS logs? If so use ULS log viewer to filter them.
What has changed in your environment? Is there a higher load or a new deployment/solution etc.
If you go to Central Admin, and check what "Services on Server" are running for each box, it would help you figure out how your services are spread throughout the farm.
Most service applications you would want to run on App servers, with the exception of Distributed Cache (app fabric) which should run on the Web Front Ends.
You are running Distributed Cache, right?
ULS log viewer is good for checking SharePoint logs (it doesn't check IIS logs though).
Event viewer on the Servers will also give you some hints.
Task Manager can show you the processes that are using the most resources on the server... Ones to look out for:
