5

This comes up in the crawl logs:

Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled.

Now, this happens for a bunch of sites that only have one thing in common, they're static HTML.

This isn't a loopback problem, this happens no matter what site I point it at. It shouldn't be a content-access account problem because, hey, these are public-facing sites. What's going on?

Improve this question
8
  • Can you check the ULS logs? No hint there like 407 Proxy authentication required? No group policy blocking your content access account on some content security servers?
    – Alexey Krasheninnikov
    Jun 2, 2011 at 10:46
  • Good point, ULS is the place to start. This is doing my head in though. Here's two of the sites I'm trying to index: asia-pacific.undp.org and undp.org
    – Dan
    Jun 2, 2011 at 11:13
  • You certainly can crawl bog standard internet sites - I've seen more than one dev can a dev server by trying to index the whole damn Internet! Check the ULS logs for HTTP response codes as Alexey suggests. I suspect a networking issue preventing the index server reading those sites.
    – Andy Burns
    Jun 2, 2011 at 11:17
  • Wow, if I could index the whole internet imagine how popular I'd be at work! We're behind ISA2006 and use Symantec security. Could it be Symantec buzzing off something sending out dozens of HTTPs? Or an unknown agent?
    – Dan
    Jun 2, 2011 at 11:27
  • 1
    Here's the IIS logs for a server in our DMZ: 2011-06-03 08:26:33 W3SVC470963155 58.181.195.7 GET / - 80 - 172.19.92.114 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 401 1 0 2011-06-03 08:26:33 W3SVC470963155 58.181.195.7 GET / - 80 - 172.19.92.114 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 401 1 2148074252 2011-06-03 08:26:33 W3SVC470963155 58.181.195.7 GET / - 80 - 172.19.92.114 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 401 1 0
    – Dan
    Jun 3, 2011 at 8:33

2 Answers 2

Reset to default
4

HA! Nailed it.

I had a look at the logs of an IIS box in our DMZ that was showing the same activity. It turns out in IIS anonymous access was turned on (of course) as well as Windows Integrated Authentication. So what's happening is that the spider is trying to use its credentials (which are no good on this machine as it's not on the domain) instead of requesting the pages anonymously. If I turn off integrated authentication it indexs OK. So I guess I just change the content access account or some such for that content source.

Improve this answer
1
1

The service account by your crawler may not have any needed proxy settings configured.

Have a look in your ULS or Windows Application Event Logs for details.

Improve this answer
4
  • This could be a comment and this is something that was suggested 40 mins back by another user.
    – Shoban
    Jun 2, 2011 at 12:09
  • Hi @Shoban, Comments can't be marked as answers so I summarised what I thought to be the most likely causes.
    – Jeroen Ritmeijer
    Jun 2, 2011 at 14:41
  • Agreed the downvote was not necessary for an answer. Sure it could have been a comment but considering the causes for the issue this seems a reasonable answer. It doesn't need to be accepted by the OP but its a viable answer. Upvoting to 0.
    – MichaelF
    Jun 2, 2011 at 15:15
  • Upvoting to 1, as the issue tackled in the answer described here can cause exactly the issue the question is asking.
    – James Love
    Jun 3, 2011 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.