I am looking for a web Crawler that works correctly with MOSS 2007 and SP 2010. Basically, I want this Crawler to grab the Sharepoint (MOSS 2007 and SP 2010) site and store it locally. This web Crawler (Also called "web robot" or "web spider") grab web pages (including resources like images and CSS…etc), download them locally and adjust any resource hyperlinks to point to the locally downloaded resources.

I found some resources, samples and tools. But I didn’t see any one specific for SharePoint. Besides, the main problem that I am facing is how to specify the depth level of Crawling?

For example: I have a SharePoint Web Application that consists of 3 SiteCollections and each SiteCollection contains 9 levels of SubSites. Assume that I only want to crawl 5 levels of the SubSites of the second SiteCollection. Well, of course the URL of that exact SiteCollection should be provided, but then, How can I crawl only 5 levels of the SubSites?

I am not asking about developing a web part for this purpose. I am asking about: How can I create a web application that is capable of accessing and crawling any online SharePoint Site (MOSS 2007 or SP 2010) according to depth level?

I would greatly appreciate any input !


Have you had a look at SharePoint Workspace 2010 to save local copies of content?

  • Thanks James for posting. But actually I want kind of a generic Crawler tool that works for the online Sites of SP 2010 and MOSS 2007. That means I am not allowed to access the SharePoint environment at all, The Crawler tool should only take 2 paremeters: URL and depth level. So, any idea ? – Alex Aug 23 '10 at 13:54
  • Considering most of the functionality of any SharePoint site is dynamic and relies on server-side code, does the site you intend to create a local copy of have purely static content (such as a publishing site)? – James Love Aug 23 '10 at 14:54
  • Exactly James! I want to create a local copy of SharePoint site(s) that are purely static. And my main problem is crawling and generating static copy based on the subsites levels. How this could be done ? – Alex Aug 24 '10 at 7:33
  • Have you tried using a bog-standard offline website browser such as HTTrack to see what would happen? httrack.com – James Love Aug 24 '10 at 8:02
  • Yes, I checked it but it doesn't fit my requirements. My case is different. I want this Crawler to be very customized and deals with depth levels. Also, there are other features and functionalities should be integrated altogether in the same application. – Alex Aug 24 '10 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.