SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have some external sites that I want SharePoint to index. There are some links that I need crawled that are not initially visible on the site because they are loaded via Ajax. I have added these links to both the sitemap.xml files and also the urllist.txt file, but SharePoint doesn't seem to recognize or crawl the links on these files.

How can i get SharePoint to work in this way?

UPDATE : Just to clarify this external site is just a website, it isn't a SharePoint site. Originally SharePoint had given me a message that it was ignoring the sitemap.xml because of a no-follow meta. The only place I thought this could be was in the robots.txt file, so I removed it but I still am not having luck.

share|improve this question

Where do you store these links? If in a list, make sure the list is marked with Searchable in the Advanced Settings.

Run a crawl after changing these properties. Putting it in the sitemap doesn't really help.


share|improve this answer
Thanks for the reply. The external site isn't a SharePoint site. It's just a regular website. It's crawling all the other pages ok, it's just not crawling the links in the sitemap.xml or urllist.txt file which I thought the SharePoint crawl engine did. – Doug Apr 3 '12 at 19:30
Ah, ok, then probably your problem comes from the actual file extensions that are considered in the crawling (under Crawlin - File Types) in the Search Service Application. – Marius Constantinescu - MVP Apr 3 '12 at 19:55
xml and txt are both listed under "File Types". Is there somewhere else I need to configure them to follow the links? – Doug Apr 3 '12 at 20:03

Try to set up the content source at HTML web site and not SharePoint Site, then indexer will behave as bing/google.

Disadvantage with this method is that scheduler always will run a full crawl.

share|improve this answer
I have the content source set up as "Web Sites". – Doug Apr 3 '12 at 20:00

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.