All crawling item return Access is denied for SP2016 single server

Question

I have setup a new SP2016 single server installation for test purpose. Regardless the best practice, I use the setup account as the crawl account for my Search Service Application.

After first time full crawl, it return the "Access is denied. Verify that either the Default Content Access Account has access ..." error. I checked the web application user policy and it has full read. I tried login within server and success browse to all pages. In crawl log, it returns 1 error and 750 success. The 750 success are the 750 users's mysite. The 1 error is http://servername. I only have 1 web application and 1 site collection http://servername.

I tested search with "People" scope result is correct. However none is returned for "everything" scope.

I have also:

• Change registry for DisableLoopBackCheck and BackConnectionHostNames
• add the default content access account to web application's user policy with "ALL READ"
• within the server (win2016), I opened IE and login with default content access account, I can browse all SharePoint pages without issue
• delete and recreate search service application
• click on search service application > administrator > grant Full control to the default content account
• try to switch authentication method between NTLM and Kerberos (both have same error "Access is denied ...")
• Use GrantAccessToProcessIdentity cmdlet to grant the default search account full permission towards the web application in question
• Create a new content DB and move all sites (using move-spsite) to it
• try to crawl only a single site collection (http://servername/sites/project)
• create a new web application and root site collection (http://servername:81). Crawl it with same setting and return exactly same error.
• Set "Ignore SSL warnings" to Yes
• Enable anonymous access

Error always the same. No result can be crawled in main site. Always the Context: Application 'Search_Service_Application', Catalog 'Portal_Content' Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled. (0x80041205) error. Anything I missed?

Answer 1

EDIT See this solutions

Access is denied. Verify that either the Default Content Access Account has access

'Retrieve People Data for Search Crawlers' rights

disable the LoopBackRequest

Answer 2

In short, the network connection is using proxy and authentication cannot pass through. Hence the crawling return 401 access denied.

Here is how I find the cause.

1. I checked IIS raw log. The c-ip (client ip) is an unknown IP address for me. After checking it is our company's proxy server ip. Also cs-username is blank and sc-status is 401.

2. Checked IE > Internet option, connection have not setup any proxy.

3. Find this article on web. Run following command to clear the proxy settings and restart search service:

a. netsh winhttp show proxy
b. netsh winhttp reset proxy
c. Net stop Osearch16
d. Net start Osearch16

4. Crawl again and this time is success. Checked the IIS log, no more 401 error and username is my crawling account.