After doing a lot of matrix testing on some troublesome URLs, there appear to be at least three things which generate this same message.
1) There is, in fact, a bot tag in a page which is having its content crawled.
2) Your SharePoint SSA is not configured to crawl the file type reflected in the URL. We are using a custom protocol handler to crawl in database information and then pass a destination URL. (Imagine that we were crawling a client list, and then making www.your_big_client.com as the destination when you searched the client name.) In our case, even though the actual URL was not being crawled, SharePoint was parsing the URL to see if it was of a valid document extension – in this case ‘.wl’. In some of my dummy URL’s, I was feeding in .asmx files and these were generating the same error. You will need to go to File Types in the SSA, and add your oddball extensions. You may need to restart the Search Service.
3) A crawl rule is in place which prevents crawling. You may specify crawl rules in order to avoid crawling particular domains, machines, page names, etc. In previous versions of SP, this had its own distinctive message, but I see evidence that crawl rule exclusions now trigger the above message. Go to Crawl Rules in your SSA, and input the offending URL to see if it matches an exclusion rule.