The item was deleted because it was excluded by a crawl rule

Question

My search crawl is not working for my local site. On my Crawl Log I see a warning:

Item not crawled due to one of the following reasons: Preventive crawl rule; Specified content source hops/depth exceeded; URL has query string parameter; Required protocol handler not found; Preventive robots directive. ( This item was deleted because it was excluded by a crawl rule. )

Important information:

• I just moved all my content databases to a newly installed Sharepoint on a new server
• I am running Sharepoint Server 2019 Standard
• When I change the address in the content source to full FQDN, the search works partially with many 401 warnings
• The address in question is https
• Same address with sps3 gets crawled successfully with no issues
• There is no robots.txt file in the site

I tried and checked everything I could think of, including:

• I checked that the default access account has access to the website
• I browsed the website ( using a browser ) while being logging in using the default access account
• I made sure there are no crawl rules
• I created a new crawl rule to include the affected URL
• I did "index reset"
• I disabled loopbackCheck
• I used Fiddler to monitor the traffic, the crawler is not even accessing the site

If you have any suggestions, please advice.

Answer 1

After two days of trying and failing, I solved the issue by cleaning up the content databases from missing features.

I had to manually uninstall all missing features using:

stsadm -o uninstallfeature -id <GUID> -force

After doing that the crawler started crawling again. But I don't know which feature in particular was causing the issue.