Tell me more ×
SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. It's 100% free, no registration required.
up vote 2 down vote favorite

I am currently configuring the search crawler to access our existing non-SharePoint document management system in order to include these documents in Enterprise Search. The document management system requires a username and password to access and I have created an account that has minimal privileges.

However, the document management system has some links which perform actions, for example bookmarking a document. If the crawler accesses these, it will perform the action for the crawler's account. Eventually, this will mean that the crawler will have performed the bookmark action on every single item within the document management system, creating significant database bloat.

It is possible to exclude URLs from the Crawl Configuration. However, does this prevent the crawler from accessing the URL or does it still access it to find more links, but just exclude that URL from the search result?

share|improve this question

1 Answer

up vote 2 down vote accepted

Did you try creating a Search Crawl rule? You have a setting in crawl configuration to "Follow links on the URL without crawling the URL itself". This option would not crawl the URL directly and should probably serve your purpose. Try it out.

enter image description here

share|improve this answer
This seems to have done the trick, I was slightly worried that it would still go to the excluded URL to find more links, but not index it. Exclusion completely prohibits it from going there at all it seems, exactly what I want. In some cases, I do want it to visit a URL, but not index it in which case creating a Include rule but checking "Follow links on the URL without crawling the URL itself" does the trick. Thanks for your help. – KingJ Jan 12 '12 at 16:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.