I have some external sites that I want SharePoint to index. There are some links that I need crawled that are not initially visible on the site because they are loaded via Ajax. I have added these links to both the sitemap.xml files and also the urllist.txt file, but SharePoint doesn't seem to recognize or crawl the links on these files.

How can i get SharePoint to work in this way?

UPDATE : Just to clarify this external site is just a website, it isn't a SharePoint site. Originally SharePoint had given me a message that it was ignoring the sitemap.xml because of a no-follow meta. The only place I thought this could be was in the robots.txt file, so I removed it but I still am not having luck.

2 Answers 2


Where do you store these links? If in a list, make sure the list is marked with Searchable in the Advanced Settings.

Run a crawl after changing these properties. Putting it in the sitemap doesn't really help.


  • Thanks for the reply. The external site isn't a SharePoint site. It's just a regular website. It's crawling all the other pages ok, it's just not crawling the links in the sitemap.xml or urllist.txt file which I thought the SharePoint crawl engine did.
    – Doug
    Commented Apr 3, 2012 at 19:30
  • Ah, ok, then probably your problem comes from the actual file extensions that are considered in the crawling (under Crawlin - File Types) in the Search Service Application. Commented Apr 3, 2012 at 19:55
  • xml and txt are both listed under "File Types". Is there somewhere else I need to configure them to follow the links?
    – Doug
    Commented Apr 3, 2012 at 20:03

Try to set up the content source at HTML web site and not SharePoint Site, then indexer will behave as bing/google.

Disadvantage with this method is that scheduler always will run a full crawl.

  • I have the content source set up as "Web Sites".
    – Doug
    Commented Apr 3, 2012 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.