Who exactly is crawling my site?

"I've decided to block all crawlers to the site other than Google or Bing"

And this is why I find people that respond to privacy complaints about Google with "if you want to switch search engines no-one is stopping you" frustrating.

Additionally, I would like to point out that according to those numbers, there are ~41,000 (418,814 - 199,725 - 40,359 - 36,340 - 33,893 - 26,325 - 13,458 - 10,657 - 6,109 - 5,993 - 4,959) additional robot hits, many of which generally are bots that only visit one or two pages. For comparison, that's more requests than Googlebot did. Just flat-out banning everything frustrates people, and encourages them to just ignore robots.txt entirely.

If you're being hammered by a bot, contact the bot's owner! Most bots have a link in the user agent that you can follow. Barring that, ban the specific bot. But don't ban everything just because a few (proximic and ADmantX) are hammering the site.

Your assumption is wrong. Ahrefs is a backlink compiling bot that tracks, documents and records in/outbound links. They do not sell services directly.

So, you have a problem with Ahrefs, too? But what about Ezooms? That's a very annoying bot for my site, drives zero traffic. Also, what about Cyveillance? They show up with a lying User-Agent field ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)"), but run Linux, and they ignore robots.txt by not even asking for it.

I actually blocked Ahrefs, Ezooms, Yandex and Cyveillance by IP address range in httpd.conf for a while, but I've decided to send them randomly-generated HTML (http://stratigery.com/bork.php) instead. I'm really surprised by Ahrefs and Ezooms appetite for gibberish about naked celebrities, condiments and sweater puppies.

I was just going by this page: https://ahrefs.com/pricing_plans.php

It is possible I am misinterpreting what they are offering.

How about using Crawl-delay at robots.txt?

http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...