Background
We are in the middle of upgrading our SharePoint farm from 2010 to 2013, and I am running into some issues with search content. In 2010, we utilized FAST Search for SharePoint 2010, and I designed a few .Net content connectors for our external systems. This proved to work extremely well.
Most of the content sources continue to work well in 2013, aside from the DMS connector (by far, the largest - ~13 million items). The full crawl completes as expected, presenting roughly a 7-8% error rate due to resource constraints, encrypted files, etc. This is where the trouble starts.
Problem
In 2010, these errors are recrawled to great success. Within a few incrementals, the completion time is reduced to < 5 minutes and the error count drops significantly. However, in 2013, it takes much longer to reduce the crawl rate and error count, with incremental times hanging around 45 minutes at best (this is after several days). To make matters worse, after an undetermined amount of time, the incrementals will either recrawl every item in the index or delete every item in the index.
After much research, the best connection to this issue seems to be the index deletion policies:
Since the articles uses the term item, I adjusted the ErrorCountAllowed and ErrorIntervalAllowed to the lowest possible values in hopes to immediately remove the erroneous items from the index and significantly reduce crawl times. However, as possibly could be expected, this just kicks off a full deletion with the next incremental -- I guess I misunderstood the articles.
Question
Has anyone else experienced these issues with BCS connectors?
FAST Search seems to be much better at processing content, so I did not run into these issues in 2010. Do I need to increase the deletion thresholds to ridiculous levels and just accept the reduced rate of content freshness?
Has anyone discovered a way to programatically remove erroneous items from the search index based on the crawl logs?
Thanks for your time and attention.
