Who exactly is crawling my site?

I see crawlers as enablers - I've even gone so far as spinning up a dedicated server instance to serve content to the bots - so they get good response times which increases relevancy in search results.

So I'm dealing with this now too, although at about 10x the traffic mentioned in the post. The first place I started was using the 5G Blacklist/Firewall [http://perishablepress.com/5g-blacklist-2013/] which is really just a great set of .htaccess rules for blocking known bad bots. Legit bots will respect your robots.txt, so if one (looking at you Yandex) is getting too aggressive, slow them down with the 'Crawl-delay: 60' (time in s) directive. Of course rogue bots don't respect this, so they get added to the blacklist rules based on UserAgent.

What I've discovered though is that bots are not my biggest worry; it's the scrapers that are stealing my client's content and re-posting it. We've successfully filed 5 DMCA complaints at this point which have been effective at stopping the known offenders, but the crawlers continue and new copycat sites keep popping up. I've found that running a 'grep Ruby access_log' returns a good chunk of the offending crawlers (not just Ruby, also search for Python and Java). Running 'host (ip address)' almost always traces back to AWS. These log entries also very rarely list a referrer.

Obviously not all of the grep results are malicious. A little research can reveal the IP is linked to a service or co you want crawling your site. For those that are unknown, they usually get an IP ban until I can determine otherwise (which the client is totally OK with, they don't want their content re-published off-site).

I've thought about setting up a honeypot, but my issue was keeping the bait links hidden from legit services (plant a hidden link on the page that only a bad actor would follow, then trap their IPs in the log and ban them). Since the DMCAs have been so effective, I haven't been forced to pursue a honeypot, but I would be very interested if anyone else has a good solution.

I also discovered that image hotlinking was a _HUGE_ problem with this site. The poor site had been neglected for years and hotlinkers were running wild. Shutting that down with a simple htaccess rewrite rule really helped. That is also how I discover content thieves, 'grep 302 access_log' and look at the referrer URLs.

Fortunately for us new(er) search engines (I work at blekko), only a small fraction of websites actively block crawlers from new search engines. It's a chicken and egg problem: we're going to have a big index for a long time before we have big traffic.

I also have a bunch of hobby websites, and I agree that bing crawls a lot more than Google and sends me a lot less traffic. And Yandex and Baidu have very little English-speaking usage, so they aren't sending me much traffic.

We ( http://samuru.com ) limit our crawling, and we honor robots.txt We also use Google Infrastructure, so we come from the same IP's as Google Bot.

We probably crawl 50 pages for every visitor we deliver. There are a lot more pages on the web than people on the planet. So the ratios will always favor the bots.

PUsH: http://en.wikipedia.org/wiki/PubSubHubbub

Google is the only crawler I see using the protocol on my site. It does make Google updates occur within minutes, so that alone is reason to implement push.