I am not able to crawl website that has STS (security token service) based authentication using FAST Search for SharePoint 2013 . I tried Form credentials and Cookie based authentication in Crawl Rules but following error comes:

Error: Access is denied. You may not have sufficient privileges to perform the operation.

What is the right way to crawl an external web that is secured with STS based authentication?


You need to create a new crawler rule with default crawler account if it already has read permission on external websites, if not then you can specify a new crawler rule with a crawler account that has at least read permission on external content your crawler will crawl.

Since you getting Access denied error seems like your crawler account doesn't have permissions to crawl external websites regardless of authentication type.

Other then that,

Ensure that the domain account that is used for the default content access account or any other content access account is not the same domain account that is used by an application pool associated with any Web application that you crawl. Doing so can cause unpublished content in SharePoint sites and minor versions of files (that is, history) in SharePoint sites to be crawled and indexed.


Hope it helps.

  • As I already mentioned, I tried Form based and Cookie based crawler rule. In both cases, I specified an account that has read permission on external content. When we make crawler rule it authenticate account credentials with external sites and if it is successful then it creates rule. In my case when I created rule, account credentials were successfully authenticated with external content and than rule was created. Note that there is Test option on Crawl Rule page, that takes URL as input and highlight which rule is applicable. Test is also passed. – Jamil Feb 15 '13 at 5:18
  • I saw following text on your source link: Source "If you are using claims-based authentication, make sure that that Windows authentication is enabled on any Web applications to be crawled." Does it mean Windows Authentication must be enabled on external website if it uses claim based authentication for crawling? – Jamil Feb 15 '13 at 6:01

I explored web in order to solve this issue but it seems it is not possible to crawl an external website using SharePoint search that has STS based authentication and don't have Windows authentication enabled. Here are few references:

"If you are using claims-based authentication, make sure that that Windows authentication is enabled on any Web applications to be crawled." Source

"SharePoint can only crawl using a Windows NTLM authentication account. Your content source must authorize the Windows account sent as part of the crawl request in order to access the document content. Though claims authentication is supported in SharePoint 2010, the gatherer is still not a claims-aware application and will not access a content source that has claims authentication only." Source

"No, it needs NTLM/Kerberos in order to crawl the Web App." Source

  • Your first source is great. Your second two sources are for SharePoint 2010. Since there was a big change in authentication recommendations with the 2013 wave, I wouldn't rely on those two for this answer. – Tom Resing Aug 21 '14 at 14:50

There is one more TechNet reference for SharePoint 2013 which supports the need for Windows NTLM for Search Crawling.

Jamil already quotes Plan crawling and federation in SharePoint Server 2013 from TechNet's SharePoint 2013 Authentication Planning.

Here's another reference from a different page in the same section:

The crawl component requires NTLM to access content. At least one zone must be configured to use NTLM authentication. If NTLM authentication is not configured on the default zone, the crawl component can use a different zone that is configured to use NTLM authentication.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.