I am seeing thousands of warnings in my crawl log for a login page for a public website that is using a complex URL.

Is there any benefit then to create an exclusion crawl rule for this login page specifically?

Will search crawl faster or not take a long if I use the exclusion crawl rule for the login page?

2 Answers 2


By default, the crawler will not follow complex URLs, which are URLs that contain a question mark followed by additional parameters — for example, http://contoso/page.aspx?x=y. If you enable the crawler to follow complex URLs, this can cause the crawler to gather many more URLs than is expected or appropriate. This can cause the crawler to gather unnecessary links, fill the crawl database with redundant links, and result in an index that is unnecessarily large.

Performance will improve if you create exclusion rules. And it is one of the best practice



Exclusion rules are useful and help in performance.

Avoid crawling less relevant content by excluding one or more URLs. This also helps to reduce the use of server resources and network traffic, and to increase the relevance of search results.Creating crawl rules to exclude URLs is particularly useful when planning start addresses for external content because the impact on resource usage is not under the control of people in your organization.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.