I have an intranet site (non-SharePoint) that is located at a site like http://www.oldportal.example.com. The site is hosted on an Apache server and is only accessible on the network. The site uses no authentication method.
I also have a SharePoint 2010 Foundation farm on the same network located at https://newportal.example.com. This environment also has Search Server Express.
I created a content source that includes http://www.oldportal.example.com/index.php with "Only crawl within the server of each start address" selected in crawl settings. I also created an include crawl rule for http://.oldportal.example.com/
Until all the content and solutions on http://www.oldportal.example.com can be migrated to https://newportal.example.com it is neccessary that the content source spider the old site.
When the crawl completes I see an "Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled." error. Am I messing up some configuration setting here, or is there an access problem I am not seeing?
Side note - this crawl has worked before, albeit in a limited capacity.