I have some external sites that I want SharePoint to index. There are some links that I need crawled that are not initially visible on the site because they are loaded via Ajax. I have added these links to both the sitemap.xml files and also the urllist.txt file, but SharePoint doesn't seem to recognize or crawl the links on these files.
How can i get SharePoint to work in this way?
UPDATE : Just to clarify this external site is just a website, it isn't a SharePoint site. Originally SharePoint had given me a message that it was ignoring the sitemap.xml because of a no-follow meta. The only place I thought this could be was in the robots.txt file, so I removed it but I still am not having luck.