Custom BCS Connector Incremental crawl not working with AssociationNavigator

Question

I created a custom BCS connector in SharePoint 2013.
I need to use the changelog approach for incremental crawling. Since I am crawling the file system like data source I am using AssociationNavigator sice it has a parent/child model. I am having the same exact issue as mentioned here http://social.technet.microsoft.com/Forums/systemcenter/en-US/34205590-71e8-4a6d-ab85-4522ff8cb199/bcs-multiple-entities-associationnavigator-incremental-crawl?prof=required

The problem seems to be when using AssociationNavigator the ChangedIdEnumerator and DeletedIdEnumerator are never called because I am using AssociationNavigator.

Does anyone have any idea how I can you changelog incremental approach when using AssociationNavigator?

Thanks!

Answer 1

I'm having the same issue with my custom BCS indexing connector for SharePoint 2013. Although it's not reflected in the official documentation, here's what I'm reading in the code sample from Microsoft (https://code.msdn.microsoft.com/SharePoint-2013-MyFileConne-79d2ea26/sourcecode?fileId=60656&pathId=390205958):

Entity

Incremental crawl: If the Entity is the source for any AssociationNavigator methods that are marked with the 'DirectoryLink' property, the behavior is the same as in the full crawl. If there are no AssociationNavigator methods where the Entity is the source, marked with the 'DirectoryLink' property, and the Entity has both a ChangedIdEnumerator and DeletedIdEnumerator defined, then both of those methods will be called in an incremental crawl. If none of the above is true, the incremental crawl behavior is the same as full crawl.

So it seems you cannot have changelog-based incremental crawls for your "parent" entities.

And for child entity instances:

Non-container-type entity instance: Full and incremental crawl behavior is the same.

I don't know how authoritative the source of this information is though.

Answer 2

The comments in that example do the best job of describing the crawl process that I've seen. The section that describes the incremental crawl BCS Model configuration is just below the section referenced by johnnyjob. The relevant bit is:

Incremental crawl: This is the same as in a full crawl, unless several things are configured to enable your shim to more intelligently decide what to emit. Here's what needs to be configured to enable this behavior:

1. The SpecificFinder of the Entity has to return a field containing the number of deleted direct child items in that particular container and this field must be identified in the model file by defining the 'DeletedCountField' property on the SpecificFinder method instance, where its value must be the name of the field that returns the delete count.
2. The AssociationNavigator for which this Entity is the SourceEntity must include in its return type descriptor a last modified time DateTime field and this field must be identified in the model file by defining the 'LastModifiedTimeStamp' property on the AssociationNavigator method instance, where its value must be the name of the fiels that returns the last modified DateTime.
3. The AssociationNavigator for which this Entity is the SourceEntity must have an input filter defined, and that input filter must have the 'CrawlStartTime' string property defined:

<FilterDescriptor Name="LastModifiedFilter" Type="Input">
   <Properties>
      <Property Name="CrawlStartTime" Type="String">x</Property>
   </Properties>
</FilterDescriptor>

4. The AssociationNavigator for which this Entity is the SourceEntity must take a DateTime input parameter, and that parameter must be associated with the filter described in #3:

<Parameter Name="lastModifiedTime" Direction="In">
   <TypeDescriptor Name="lastModifiedTime" TypeName="System.DateTime" AssociatedFilter="LastModifiedFilter" />
</Parameter>

If this is the case, then the association navigator is called, and your shim will be provided the last modified time to determine what items to return.

I've implemented this in a BCS model for the SQL Connector and it is working.