So I have a sharepoint 2010 server farm. I have a separate custom intranet that I want to have sharepoint crawl. The custom intranet is a .net website. The default page allows anonymous authentication. However clicking any other page send does a http 302 redirect to another Login page that automatically takes your login credentials, logs you in and redirects you back to the page request. One time. SharePoint search crawls the first page then fails going anywhere after that. The crawl logs give many warnings, here is one example:
The URL was permanently moved. ( URL redirected to http://customintranet}/Login/Login.aspx?ReturnUrl=%2fpoll.aspx )
I have a hunch that the crawler just dies here and 302 is the culprit. If I use my browser as the sharepoint search account it works fine. I can browse the site. Is there any trick I can do?
On the content source box where I add the links, I entered the link above thinking I could run that first, in a sense, cook the site first by avoiding the redirect and it would work and the next line would just be the root link. However it keeps saving the root link first followed by the Login link in that order on the text box. Also just paling the Login link above only in the content source text box does not help either. It still errors out.
UPDATE: I discovered that in the crawl rules you can specify form credentials. Our custom intranet does contains a login form page. I choose the form option but when I click the Get Credentials button it just hangs at the Loading dialog. Same for cookie credentials. I notice there is an option to load a cookie but I am not sure how to go about doing that. I assume cookies also have some kind of server timestamp that needs to be swapped out regularly.
Anyone ever encounter this kind of issue before?