Take the 2-minute tour ×
SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. It's 100% free, no registration required.

This comes up in the crawl logs:

Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled.

Now, this happens for a bunch of sites that only have one thing in common, they're static HTML.

This isn't a loopback problem, this happens no matter what site I point it at. It shouldn't be a content-access account problem because, hey, these are public-facing sites. What's going on?

share|improve this question
    
Can you check the ULS logs? No hint there like 407 Proxy authentication required? No group policy blocking your content access account on some content security servers? –  Alexey Krasheninnikov Jun 2 '11 at 10:46
    
Good point, ULS is the place to start. This is doing my head in though. Here's two of the sites I'm trying to index: asia-pacific.undp.org and undp.org –  Dan Jun 2 '11 at 11:13
    
You certainly can crawl bog standard internet sites - I've seen more than one dev can a dev server by trying to index the whole damn Internet! Check the ULS logs for HTTP response codes as Alexey suggests. I suspect a networking issue preventing the index server reading those sites. –  Andy Burns Jun 2 '11 at 11:17
    
Wow, if I could index the whole internet imagine how popular I'd be at work! We're behind ISA2006 and use Symantec security. Could it be Symantec buzzing off something sending out dozens of HTTPs? Or an unknown agent? –  Dan Jun 2 '11 at 11:27
1  
Here's the IIS logs for a server in our DMZ: 2011-06-03 08:26:33 W3SVC470963155 58.181.195.7 GET / - 80 - 172.19.92.114 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 401 1 0 2011-06-03 08:26:33 W3SVC470963155 58.181.195.7 GET / - 80 - 172.19.92.114 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 401 1 2148074252 2011-06-03 08:26:33 W3SVC470963155 58.181.195.7 GET / - 80 - 172.19.92.114 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 401 1 0 –  Dan Jun 3 '11 at 8:33
show 3 more comments

2 Answers

up vote 3 down vote accepted

HA! Nailed it.

I had a look at the logs of an IIS box in our DMZ that was showing the same activity. It turns out in IIS anonymous access was turned on (of course) as well as Windows Integrated Authentication. So what's happening is that the spider is trying to use its credentials (which are no good on this machine as it's not on the domain) instead of requesting the pages anonymously. If I turn off integrated authentication it indexs OK. So I guess I just change the content access account or some such for that content source.

share|improve this answer
1  
Oh dear. The medicine seems a bit tough to swallow: blog.mikehacker.net/2010/06/10/… –  Dan Jun 3 '11 at 10:21
add comment

The service account by your crawler may not have any needed proxy settings configured.

Have a look in your ULS or Windows Application Event Logs for details.

share|improve this answer
    
This could be a comment and this is something that was suggested 40 mins back by another user. –  Shoban Jun 2 '11 at 12:09
    
Hi @Shoban, Comments can't be marked as answers so I summarised what I thought to be the most likely causes. –  Muhimbi Jun 2 '11 at 14:41
    
Agreed the downvote was not necessary for an answer. Sure it could have been a comment but considering the causes for the issue this seems a reasonable answer. It doesn't need to be accepted by the OP but its a viable answer. Upvoting to 0. –  MichaelF Jun 2 '11 at 15:15
    
Upvoting to 1, as the issue tackled in the answer described here can cause exactly the issue the question is asking. –  James Love Jun 3 '11 at 9:31
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.