SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am using SharePoint Server 2007 on Windows Server 2008. I am using Search Center to crawl web data source (i.e. crawl web page from other web sites). My question is related to crawled page counters displayed for the web data source log page of Search Center.

My question is, there are 3 crawl counters displayed, successful counter, fail counter and warning counter. For each counter value, will there be any duplication urls? For example, it is reported for web data source, 1000 are crawled successfully, 10 failed, no warning. Does it mean there are 1000 distinct web pages stored in Search Center? I am not sure whether there are any duplicated Urls in the 1000 counted pages?

BTW: I have this confusion because I set daily incremental page crawl, for example, if is crawlered both yesterday and today (both cases are successful crawl), will it be counted twice? Appreciate if anyone could provide some documents about what are the counters' meaning?

thanks in advance, George

share|improve this question
up vote 1 down vote accepted

If you crawl a regular website it is going to follow each of the links. It shouldn't duplicate pages, but it will see the reference to the home page for example many times. Ultimately you would determine the number of pages or items by looking at the Items in Index count not the number of items crawled.

share|improve this answer
Thanks Mike! Do you have any recommendations for documents that prove there is no duplication? – George2 Jul 11 '10 at 7:28
I need to generate a report about how many Urls are already crawled and if there could be a document which could prove there is no duplication, it will make me more confident. Thank you! – George2 Jul 11 '10 at 7:30
Any updates to my question above? Thank you! – George2 Jul 14 '10 at 16:16

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.