Does anyone know how to set up a robots.txt file to allow the sharepoint crawler to index content on a sharepoint site, but disallow all other bots (like google, yahoo, etc...)? Or in other words, what user agent does the sharepoint crawler use to identify itself?
1 Answer
Did not know this, but after a 2 second Bing I found this: http://www.wssdemo.com/Lists/Resources/DispFormOld.aspx?ID=2226
answered Aug 28, 2010 at 7:43
