Anyone have a recommendation for an open source web crawler that has worked well with sharepoint sites and subsites. Currently trying norconex but want to know if anyone has any experience with anything else?
2 Answers
My understanding is that third party search crawlers aren't supported for SharePoint Online. I think the main reason would be throttling and additional strain on the SharePoint Online service. SharePoint already has a comprehensive search crawler, so I wouldn't recommend using a third party crawler anyway.
There are multiple APIs for accessing the out-of-the-box search index:
answered Aug 17, 2023 at 17:07
Have you encountered any specific problems with Norconex or what are your requirements for the web crawler? How many documents should be indexed? Any requirements regarding the index freshness? Which search engine are you targeting?
Do you want to index restricted content or are the documents "public" for all users of your search engine? SharePoint has multiple layers of access restrictions. Even individual sites can contain dynamic content of witch some parts are restricted to a limited audience. I'm pretty sure that it's impossible to reflect this with a normal webcrawler.
API rate limits can become a challenge when you want to index large SPO instances but for a small site you might be lucky if you reduce the requests per minute to 30 or 60.
Medium to large instance require correct handling of SPOs response codes/headers and advanced strategies to index the content and keep it fresh. There may be no way around a professional solution to keep an index up to date.
Disclaimer: I work for a company that develops enterprise search connectors for various source systems and search engines. We have been in this business for many years and know very well the obstacles to indexing content from SharePoint Online and many other systems.
