Crawling fails with 404 error message, but the IIS log tell a different story

Question

In our clean SP2010 install, crawling stops immediately, just logging a single error message:

The object was not found. ( Error from SharePoint site: HttpStatusCode NotFound The request failed with HTTP status 404: Not Found. )

Looking at the IIS log does, surprisingly, not confirm this. Here are all the log entries related to crawling:

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2010-09-01 15:20:42 127.0.0.1 GET / - 80 - 127.0.0.1 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 401 1 2148074254 15
2010-09-01 15:20:42 127.0.0.1 GET / - 80 0#.w|global\cre-nhk 127.0.0.1 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 200 0 0 46
2010-09-01 15:20:42 127.0.0.1 GET /Pages/Forside.aspx - 80 0#.w|global\cre-nhk 127.0.0.1 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 200 0 995 937
2010-09-01 15:20:43 127.0.0.1 GET /_vti_bin/sitedata.asmx - 80 - 127.0.0.1 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 401 1 2148074254 0
2010-09-01 15:20:43 127.0.0.1 GET /_vti_bin/sitedata.asmx - 80 0#.w|global\cre-nhk 127.0.0.1 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 200 0 0 15

As you can see, just a fairly normal handshaking of 401’s and 200’s.

Finally, here are the ULS log entries:

09-01-2010 17:36:15.36  mssdmn.exe (0x0E9C)     0x1E24  SharePoint Server Search        Indexing        fsa0    Monitorable     GetVirtualServerPolicy fail. error 2147750401, strStsUrl http://demo.dsr.dk     
09-01-2010 17:36:15.36  mssdmn.exe (0x0E9C)     0x1E24  SharePoint Server Search        PHSts   dvt6    High    SetSTSErrorInfo ErrorMessage = Error from SharePoint site: HttpStatusCode NotFound The request failed with HTTP status 404: Not Found. hr = 80041201  [sts3util.cxx:5095]  d:\office\source\search\native\gather\protocols\sts3\sts3util.cxx   
09-01-2010 17:36:15.36  mssdmn.exe (0x0E9C)     0x1E24  SharePoint Server Search        PHSts   dvi3    High    ***** Couldn't retrieve server http://demo.dsr.dk policy, hr = 80041201         [sts3util.cxx:1738]  d:\office\source\search\native\gather\protocols\sts3\sts3util.cxx  
09-01-2010 17:36:15.36  mssdmn.exe (0x0E9C)     0x1E24  SharePoint Server Search        PHSts   dvu0    High    STS3::StoreCachedError: Object initialization failed.  Message:  "Error from SharePoint site: HttpStatusCode NotFound The request failed with HTTP status 404: Not Found." HR: 80041201  [sts3util.cxx:5189]  d:\office\source\search\native\gather\protocols\sts3\sts3util.cxx
09-01-2010 17:36:15.36  mssdmn.exe (0x0E9C)     0x1E24  SharePoint Server Search        PHSts   dvg4    High    ****** Server demo.dsr.dk security initialization failed, hr = 80041201 error Message Error from SharePoint site: HttpStatusCode NotFound The request failed with HTTP status 404: Not Found.  [sts3util.cxx:1358]  d:\office\source\search\native\gather\protocols\sts3\sts3util.cxx  
09-01-2010 17:36:15.36  mssdmn.exe (0x0E9C)     0x1E24  SharePoint Server Search        PHSts   dv5x    High    CSTS3Accessor::InitServer: Initialize STS Helper failed. Return error to caller, hr=80041201  [sts3acc.cxx:1691]  d:\office\source\search\native\gather\protocols\sts3\sts3acc.cxx     
09-01-2010 17:36:15.36  mssdmn.exe (0x0E9C)     0x1E24  SharePoint Server Search        PHSts   dv3t    High    CSTS3Accessor::InitURLType fails, Url http://demo.dsr.dk, hr=80041201           [sts3acc.cxx:258]  d:\office\source\search\native\gather\protocols\sts3\sts3acc.cxx     
09-01-2010 17:36:15.36  mssdmn.exe (0x0E9C)     0x1E24  SharePoint Server Search        PHSts   dvb1    High    CSTS3Accessor::Init fails, Url http://demo.dsr.dk, hr=80041201                  [sts3handler.cxx:312]  d:\office\source\search\native\gather\protocols\sts3\sts3handler.cxx     
09-01-2010 17:36:15.36  mssdmn.exe (0x0E9C)     0x1E24  SharePoint Server Search        PHSts   dvb2    High    CSTS3Handler::CreateAccessorExD: Return error to caller, hr=80041201            [sts3handler.cxx:330]  d:\office\source\search\native\gather\protocols\sts3\sts3handler.cxx    
09-01-2010 17:36:15.36  mssdmn.exe (0x0E9C)     0x1E24  SharePoint Server Search        FilterDaemon    e4ye    High    FLTRDMN: Errorinfo is "Error from SharePoint site: HttpStatusCode NotFound The request failed with HTTP status 404: Not Found."  [fltrsink.cxx:553]  d:\office\source\search\native\mssdmn\fltrsink.cxx
09-01-2010 17:36:15.38  mssearch.exe (0x1BD0)   0x0F28  SharePoint Server Search        GatherPI        e5e0    High    advising status change NSC_FLUSH (467f7b97-cba6-4ebd-ab96-4fa7af378718-crawl-0 Portal_Content crawl -1)  [gatherobj.cxx:4512]  d:\office\source\search\native\gather\server\gatherobj.cxx      
09-01-2010 17:36:15.38  mssearch.exe (0x1BD0)   0x0F28  SharePoint Server Search        Gatherer        cd11    Warning The start address http://demo.dsr.dk cannot be crawled.  Context: Application

Things checked and verified: · DisableLoopbackCheck is disabled · The crawler user (cre-nhk) has “Full Control” on the web application (that should include “read all”) · The crawler user can log on to the site and view all pages when accessing them using Internet Explorer · There is no Alternate Access Mapping used

I see that someone else has a similar issue:

http://social.technet.microsoft.com/Forums/en/sharepoint2010setup/thread/f3c61b53-304a-4c2a-a370-d0e573219d1d

But in our case: - it's a clean install, no upgrading - Plain HTTP, no certificates

Any answer/input most appreciated!

Best regards

Thomas

Answer 1

We resolved this issue by going back to crawling using the farm admin account. This project has a custom approach to authentication, using a custom MembershipProvider, which seems to bite us again and again. So, as a general advice, avoiding authenticating that way if you can. (And had the project started anew, they probably would have chosen a somewhat different approach…)

Regards

Thomas