I have a publishing page library containing all of my posts. We are using search driven webparts to pull the search results and display to users. What we are seeing is that after a new page is added, it is not indexed during the next incremental crawls (let's say 10-15 incremental crawls have run) or several full crawls. Then, suddenly, the scheduled crawls will magically pick it up. This is problematic because there are time-sensitive posts that need to show on the site relatively quickly.
Is this a bug in sharepoint? What triggers the incremental or full crawls to pick up that a new page has been added that I am missing. The pages are are fully published and approved and none of the settings about search result removal/library settings/etc are causing this - without changing anything else, the new page will be indexed and show in the crawl logs.
Anyone have any ideas? We're still stuck on this one.