We are having SharePoint 2016 On-Premises and crawling external content in a website based on ASPX pages.

The issue here is that When doing incremental all pages are being considered as modified pages, though there is no changes in many of the ASPX pages.

Is this the expected way?

Is there any better alternative to do Incremental crawl for external ASPX pages?

Edit1: We modified the page to HTML and in HTML when we did incremental crawl - SharePoint Crawler senses correctly and mark the pages those are not modified and working perfectly.

Edit2: Found that header of the ASPX page automatically gets modified ,SharePoint considers it as modified due to that.Is there anyway to make it constant value by code or any other means.

Please advice.

  • 1
    My guess is the ASPX pages are not returning the correct headers that any crawler (and web browser/proxy) relies on to determine if the page has changed, (i.e. Etag or Last-Modified) and thus, always thinks the page is new.
    – Greg W
    Commented Jun 19, 2017 at 12:58
  • @GregW do you have any Idea that we can set the headers correctly or fixed things? Commented Jun 27, 2017 at 9:31
  • 1
    Html works as it's a static file and IIS returns the file's last modified attribute in the responses. With ASPX you will have to set the headers manually as these files are assumed to be dynamic and are always "latest". Setting the headers will depend on what application you're using.
    – Greg W
    Commented Jun 27, 2017 at 9:36
  • @GregW Do you know how i can set the headers manually in my c# code? Commented Jun 27, 2017 at 9:50
  • 1
    Sorry. I'm not in a position this second to write this up fully (will do so in a proper answer soon). There's SetLastModified (msdn.microsoft.com/en-us/library/…) and manually via AddHeader.
    – Greg W
    Commented Jun 27, 2017 at 9:57

2 Answers 2


Did some research on the cache policy for MVC and found that by default, MVC request is not cached by the IIS.

If this happens, when incremental crawl requests for an apsx page, the request will always result in a 200 Response. This is not the case for static pages. IIS will cache the HTML pages and they will result in 304 Response (the reason why your incremental crawl on HTML pages is working properly)

Try to cache the aspx pages so that incremental crawl's request will result in 304 Response when there is no change in the data of the aspx page. This will make sure that the pages which are not modified are not crawled by the Sharepoint's incremenal crawl.

If you are using MVC, try using OutputCache - https://msdn.microsoft.com/en-us/library/hdxfb6cy(v=vs.100).aspx

I had the same issue and used this approach for a successful incremental crawl. I have cached the Controller with a maximum time and used a custom value for caching. This custom value checked if there is any change for this data and decided if the request should be cached or not. Code Example for custom value caching - https://msdn.microsoft.com/en-us/library/5ecf4420.aspx

  • Wonderfully explained. I will implement this and keep you posted on any developments Commented Jun 28, 2017 at 13:58

Based on the comment discussion, it appears that the ASPX-based website is not sending the correct headers.

SharePoint, like any web crawler (or browser/proxy cache), relies on certain headers being sent back to it in response to a valid GET request (as an aside, it doesn't do this for SharePoint content.

It simply asks itself for all the changes since the last crawl as SP maintains a change history for this purpose). For this to function correctly, one of two headers need to be present, an E-Tag header (which gives a yes/no answer to the modified question. It either matches or it doesn't) or the Last-Modified header (which informs the requester "when" the content was last changed).

In your first edit, you noted that plain HTML files (known as static files as they are read directly from the web server's storage and aren't "processed" like ASPX pages) are correctly ignored when incremental crawling. This is due to IIS reading the file's LastModifiedTime attribute and sending that in the Last-Modified HTTP header automatically without user configuration/intervention.

Note, these headers are easily seen when using your browser's developer tools (or a tool such as Fiddler). If you look closely at a request for a SharePoint page, it has a Last-Modified header too but it'll be the current time unless you've enabled a form of caching (such as page or output caching).

Regarding your ASP.NET pages, if you have access to the Response object, you should be able to set this header with the Response.Cache.SetLastModified(DateTime) method. See: system.web.httpcachepolicy.setlastmodified

Once set, verify that the correct header is being sent using your browser's developer tools.

Forgot to add: Determining what the "Last Modified" time is, is up to you.

  • Thanks @GregW for detailed inputs, did follow the comments and added the code to make Lastmodified as constant value still the same , no progress any idea still? Commented Jun 27, 2017 at 11:26
  • Have you verified that the header is being sent? Comparing that to the headers sent by other websites? (you'll probably need to do two crawls as the first one will still be a full crawl to pick up the header)
    – Greg W
    Commented Jun 27, 2017 at 11:58
  • I did full crawl first and then I did incremental crawl. I checked the headers via developer tools and it's showing the updated last modified (constant value). Anything I'm missing? Commented Jun 28, 2017 at 2:43
  • Not sure. Are you saying the incremental crawl takes the same time as a full crawl or that the crawl logs aren't reporting that the pages haven't changed?
    – Greg W
    Commented Jun 28, 2017 at 2:59
  • Crawl logs reporting that the pages are modified still. Regarding the time for crawl, I tested with hundred items so there is not much difference between full crawl and incremental crawl time. Commented Jun 28, 2017 at 3:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.