Who exactly is crawling my site? • Hacker News Board

Re: Who exactly is crawling my site?

#71

Post by sergiosgc » Sat, Oct 19, 2013, 8:04 AM UTC

nostromo wrote: post #13

Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers. I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic. I feel like search engines sh…

I don't block completely, but have for a long time configured servers to react like this:

- If it's cached content, there's no limit

- If it's non cached content and it is a bot, direct to the pool of servers assigned to uninteresting bots

- If it is an interesting bot (Google, bing, a couple others), use instead a pool of servers for interesting bots

Some bots are not only non-traffic generating, they are absurdly aggressive, consuming 99% of server resources for the hour or so needed to pull the content, regular site users be damned. A configuration like this is reasonably low maintenance while limiting the damage by bots.

Re: Who exactly is crawling my site?

#72

Post by webhat » Sat, Oct 19, 2013, 10:18 AM UTC

The first thing I did after reading this was look at http://danbirken.com/robots.txt I guess he though of that.

  # (This is not the SEO site in question)

  User-Agent: *
  Allow: /

Re: Who exactly is crawling my site?

#73

Post by leephillips » Sat, Oct 19, 2013, 12:48 PM UTC

PaulHoule wrote: post #58

nostromo wrote: post #13

Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers. I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic. I feel like search engines sh…

It's actually a lot worse if your site is large since crawlers will make even more requests crawling a large site. If I see that a crawler is sending me any traffic at all, I will accept that, but if the amount of traffic is zero, I put them in robots.txt and in the IP block list. Although I try hard to make sure my images are all clean creative common images, I block the robots that are there to find copyrighted ima…

Is there a list of good IPs to block, similar to the hosts files that people pass around?

EDIT: chrsstrm mentions something below

Re: Who exactly is crawling my site?

#74

Post by [deleted] » Sat, Oct 19, 2013, 12:52 PM UTC

lostlogin wrote: post #48

pavel_lishin wrote: post #27

Earlier quoted context omitted.

Why not adjust the crawl rate based on how often the site's content changes? If DuckDuckGoogle crawls me once an hour for a week, and notices that I only updated my content twice, why not scale the crawling back to a more reasonable rate, like once a day?

This seems logical, however what about the person who updates rarely (once or twice a month) but get a lot of notice when they do. Search engines wouldn't want to miss the opportunity to get that traffic. I wonder if there is some sort of reverse process you could opt into - ie never crawl me until I ask to be crawled. Disclaimer: I know little about this sort of thing.

[deleted]

Re: Who exactly is crawling my site?

#75

Post by leephillips » Sat, Oct 19, 2013, 12:58 PM UTC

sergiosgc wrote: post #70

toomuchtodo wrote: post #23

Earlier quoted context omitted.

Could they not run webhooks where my site can call to them when to do a full or partial crawl? PubSub for web crawling, if you will.

PUsH: http://en.wikipedia.org/wiki/PubSubHubbub Google is the only crawler I see using the protocol on my site. It does make Google updates occur within minutes, so that alone is reason to implement push.

I thought this was only being used for RSS - interesting!

Re: Who exactly is crawling my site?

#76

Post by txutxu » Sat, Oct 19, 2013, 1:48 PM UTC

It's easy for a crawler to parse a robots.txt in different ways.

The article says to whitelist a few and deny everything else.

The crawler could parse that, and change the user agent to one whitelisted on further requests.

Re: Who exactly is crawling my site?

#77

Post by gabemart » Sat, Oct 19, 2013, 2:19 PM UTC

PaulHoule wrote: post #58

nostromo wrote: post #13

Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers. I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic. I feel like search engines sh…

It's actually a lot worse if your site is large since crawlers will make even more requests crawling a large site. If I see that a crawler is sending me any traffic at all, I will accept that, but if the amount of traffic is zero, I put them in robots.txt and in the IP block list. Although I try hard to make sure my images are all clean creative common images, I block the robots that are there to find copyrighted ima…

>I block the robots that are there to find copyrighted images because these ones exist to give me nothing but trouble.

I've never heard of these bots before. Where did you hear about them, and how do you block them?

Re: Who exactly is crawling my site?

#78

Post by ihenriksen » Sat, Oct 19, 2013, 2:58 PM UTC

You should add some request-rate settings to your robots.txt file to prevent robots from crawling your web site to often. Here is an example: http://www.sparkledb.net/robots.txt

Re: Who exactly is crawling my site?

#79

Post by bediger4000 » Mon, Oct 21, 2013, 5:34 PM UTC

bediger4000 wrote: post #6

So, you have a problem with Ahrefs, too? But what about Ezooms? That's a very annoying bot for my site, drives zero traffic. Also, what about Cyveillance? They show up with a lying User-Agent field ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)"), but run Linux, and they ignore robots.txt by not even asking for it. I actually blocked Ahrefs, Ezooms, Yandex and Cyveillance by IP address range in httpd.conf for a…

I'd like to note that the above is the first time I've ever referenced that URL. No links exist to it on my web site.

I just got a visit from the "MJ12Bot" (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php) asking for robots.txt, and then the URL above. So, MJ12bot reads Hacker News, or at least follows as the URLs listed in it.

Re: Who exactly is crawling my site?

#80

Post by mrgreenfur » Tue, Oct 22, 2013, 3:57 PM UTC

nostromo wrote: post #13

Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers. I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic. I feel like search engines sh…

I'm late, but for this there are companies that sit in the middle and block crawlers, spambots and known-malicious addresses. Here's one I use: http://www.incapsula.com/ So far I've been happy with it!