Tell me more ×
SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. It's 100% free, no registration required.
up vote 0 down vote favorite

I have particular problems with WFE performance during search jobs. Our farm contains 1 app server, 1 db server and 2 identical WFE servers (NLB) - really identical, checked 5 times already. However I try to distribute search components through farm (query & crawl on WFEs, crawl on app, queries on WFEs, one WFE failover only etc.) we always have one of our WFE overloaded. Since we added Project Server and one of our SC overcomming the 64K limit of ACL (known issue), inc. jobs running about 10 times slower than usual and because users are enable to work with portal on the affected server during this jobs, it is really an (annoying) issue. I'm force to plan jobs during night and this is not enough.

I know, that crawler no longer stores a copy of an index, items are being directly propagated to a query server, but how it is possible that one of them is completely overloaded during all jobs? Just only one of two? Network, drives, dbs, all other counters are "in green numbers".

The only and last thing what we did not tested yet is third server installation just for search only. And my managers don't want to hear a word about it.

[edit #2 beter perfmon counters, administrative reports]

enter image description here

One realy long crawl - all on STS4

enter image description here

enter image description here

Thank you for any advice.

share|improve this question
I'm wondering now where the bottleneck is, it might be worth your time running perfmon, putting up graphs for hardisk, cpu, memory, harddisk and sharepoint services, and looking for the correlation. – Hugh Wood Nov 19 '12 at 19:41
@HughWood see [edit #1] section please, today's screenshots from 3pm. I did not have much time to picked the right counters, so if you want to see more, please let me know – Molik Nov 20 '12 at 16:06
Are you getting a lot of crawl errors on the indexer? Network usage seems to correlate with the CPU usage, it might be worth looking at these two uniquely. If it does correlate, then need to go down the view of the network activity and looking at where it is going, looking for errors. Or if it is simply a ton of retries because of the amount of activity. I'm starting to think the network activity is bottle necking. – Hugh Wood Nov 20 '12 at 16:57
@HughWood constantly about 120 per one job (from 800 000 items), mainly returned error when attempting to download item messages. Also I cannot find system in crawl times, 31880 security updates takes 1 hour 53 minutes, 16690 security updates takes 9 minutes. I don't know if craowl log helps you, I can paste some lines here. – Molik Nov 20 '12 at 17:04
These error will dramatically decrease server performance, two fold, the extra waits for the content and the logs having to be written. It may be a permission issue with the crawler, or the crawler may be set up incorrectly. This is where you need to investigate. Perhaps some errors in your question from the logs may help pin it down. The other avenue is the security updates, what is in that 31,880 security updates that is causing so much extra time? Again I would consult the logs. (Do you have a list that has a per item security policy? That has a lot of items this will do it) – Hugh Wood Nov 20 '12 at 17:08
show 3 more comments

1 Answer

up vote 0 down vote

when you can't throw more resource at it, the best you can do is set the crawler to only run out of hours as you said. Since your WFEs are overloaded the only alternate solution is to add another server. You're in a pickle for sure.

This being impossible, I suggest breaking your site down, and performing separate crawls at different times, for example doing mysites at one time, large site collections out of hours, bunch a few others together etc etc. Splitting the sections of your farm up into separate jobs. Make sure you don't have the crawler running on higher priority will help also (This option is under the search service on the left in central admin).

Hope this helps.

share|improve this answer
We already made 3 different content sources (affected sites, MySites and the rest) and create a plan, that run all of them in different times. But 1x per day period for Project server and Finance SCs is not enough :( – Molik Nov 19 '12 at 16:43
Then all you can do is, tell the managers the servers are at their limits, the two choices are to put up with the once a day update of indexes, move to 2013 which removes this issue since it always updates the indexes, or get new hardware, just as a dedicated indexer. – Hugh Wood Nov 19 '12 at 16:47
That is the point, not all servers are overloaded, just only one of them and during the search only. Solarwinds is the proof, what my manages have in their hands. Search consumes everything what WFE has, even if we add some more resources, search consume it and time will be reduced in minutes only. And together with strange working NLB, which "I think" don't move anyone until some of resources on one server remains, the problem is here. – Molik Nov 19 '12 at 16:58
@Molik I see, are you using Fastsearch? It could be the task offload setting. – Hugh Wood Nov 19 '12 at 17:24
Unfortunately not. And SP 2013 (and its Request Management which may help us) is planned for Q2 2013 – Molik Nov 19 '12 at 17:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.