Does anyone know how to set up a robots.txt file to allow the sharepoint crawler to index content on a sharepoint site, but disallow all other bots (like google, yahoo, etc...)? Or in other words, what user agent does the sharepoint crawler use to identify itself?


Did not know this, but after a 2 second Bing I found this: http://www.wssdemo.com/Lists/Resources/DispFormOld.aspx?ID=2226

| improve this answer | |
  • Cool, trying it now in the robots.txt. – maxpower47 Aug 30 '10 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.