"The SharePoint item being crawled returned an error when attempting to download the item"

Question

I"m having a issue that I haven't found any way to resolve that. I ran crawler and then viewed crawl log, I saw "The SharePoint item being crawled returned an error when attempting to download the item". I searched on Google and couldn't resolve.

I checked Search feature in my Site collection that is using Claim-Based Authentication but I couldn't see any result there. Also, I see the value of Searchable item is 0 even though crawling worked well. I checked ULS and didn't see related to Search errors.

My Web application is using Claim-Based Auth with Kerberos. I heard from another that Search could crawl in Kerberos Web app thus I switched to NTLM but it also didn't work. I still have the error in Crawl log.

I'm using re-released June CU 2011 SharePoint Server 2010 patch.

Answer 1

Check the Windows event log for the server(s) that would handle requests from your crawl. You may see authentication errors which would help you track down the problem. In search configuration follow the link to set your crawl account and set it again, with password, even if it is already configured. Try retrieving the content directly after logging in with the crawl account. Make sure your pages are all checked in (and published and approved if publishing site). There is a reported issue if you have a core results web part on your default page (apparently - I assume you know this from your Googling).

That's all I can think of at the moment!

Answer 2

1. The URL mentioned in the content source should be in the default zone.
2. URL should be accessible without any issue inside the server.
3. Verify the SSL related errors and install the required webapplication certificates on the server.
4. Home page of the site collection should be "default.aspx"
5. Home page should be accessible without any issues.
6. Avoid placing core results webpart on homepage.
7. Make sure that all custom webpages are checked in.
8. Try installing Aug CUs.

Answer 3

Your Search app might need to be switched to Claim ACLs too:

Is your Default Content Access account configured and does it have full read access to the webapps you are trying to crawl?

You could also try the following:

$SearchApp = Get-SPServiceApplication SearchAppName $SearchApp.setproperty("ForceClaimACLs", 1)

This command is undocumented for as far as Google knows, but the implications seem quite clear: the ALCs are now stored as Claims instead of NT tokens.

The article is about enabling search for 1-way domain trusts, but i can imagine that if the source is using claims, the search needs to use it too? But I could be wrong about this and the PoSH command above is irreversible!

Answer 4

I would set up an internal site, with Windows authentication, just available for the crawler.

You may have http://mySPSite with Claims or whatever authentications and extension of if (as Access Mapping for an Intranet ou Custom Zone) whereas you only give access to the crawler wich in turn is configured with a Domain or Local User Account (if standalone).

The crawler will successfully index results for both Zones and Authentication issues will not occur.