I am currently configuring the search crawler to access our existing non-SharePoint document management system in order to include these documents in Enterprise Search. The document management system requires a username and password to access and I have created an account that has minimal privileges.
However, the document management system has some links which perform actions, for example bookmarking a document. If the crawler accesses these, it will perform the action for the crawler's account. Eventually, this will mean that the crawler will have performed the bookmark action on every single item within the document management system, creating significant database bloat.
It is possible to exclude URLs from the Crawl Configuration. However, does this prevent the crawler from accessing the URL or does it still access it to find more links, but just exclude that URL from the search result?