Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who exactly is crawling my site?

danbirken.com

Post reply on HN
Print view View original

31–40 of 80 posts

Re: Who exactly is crawling my site?

#31

Post by yogo »

dumbfounder wrote: post #16

If everybody does this then a new search engine will never come to be. You better hope Google is the best search engine anyone could have ever created. As a search engine developer that has tried to compete with Google in the past, I find this disheartening. I agree with blocking anyone doing bad. But you are cutting out the good with the bad.

Blocking is never a sustainable strategy but sometimes you have to kill some good cells with the cancerous ones. Chalk it up to collateral damage. It's the same situation with forum/comment spam often the IP falls within a range used by OVH so when it gets out of hand I block the entire IP block.

Re: Who exactly is crawling my site?

#32

Post by victorf »

nostromo wrote: post #13

Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers. I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic. I feel like search engines sh…

Add a time limit to robots.txt and block anyone who violates it. Maybe also add a nonexistent honeypot (I disallow /wpadmin on my static website haha). I've been fail2banning a lot of shitbirds and my traffic logs before and after are quite different.

Re: Who exactly is crawling my site?

#33

Post by breadtk »

ScottWhigham wrote: post #14

I actually did this exact robots.txt for my site this summer. The net effect? A massive loss of secondary market traffic - enough that, after a month, we probably lost 10-15% in revenue. It surprised us. We rolled the old robots.txt back into place and bam - life went back to normal. You can question my pithy 300-character explanation all you want but I'll leave you with this: we were 100% certain that the robots.txt…

Do you mind posting your old robots.txt?

Re: Who exactly is crawling my site?

#35

Post by carsonreinke »

bediger4000 wrote: post #6

So, you have a problem with Ahrefs, too? But what about Ezooms? That's a very annoying bot for my site, drives zero traffic. Also, what about Cyveillance? They show up with a lying User-Agent field ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)"), but run Linux, and they ignore robots.txt by not even asking for it. I actually blocked Ahrefs, Ezooms, Yandex and Cyveillance by IP address range in httpd.conf for a…

Block Yandex?

Re: Who exactly is crawling my site?

#36

Post by ScottWhigham »

breadtk wrote: post #33
ScottWhigham wrote: post #14

I actually did this exact robots.txt for my site this summer. The net effect? A massive loss of secondary market traffic - enough that, after a month, we probably lost 10-15% in revenue. It surprised us. We rolled the old robots.txt back into place and bam - life went back to normal. You can question my pithy 300-character explanation all you want but I'll leave you with this: we were 100% certain that the robots.txt…

Do you mind posting your old robots.txt?

It was almost identical to his

Re: Who exactly is crawling my site?

#37

Post by csharp_gooru »

ScottWhigham wrote: post #14

I actually did this exact robots.txt for my site this summer. The net effect? A massive loss of secondary market traffic - enough that, after a month, we probably lost 10-15% in revenue. It surprised us. We rolled the old robots.txt back into place and bam - life went back to normal. You can question my pithy 300-character explanation all you want but I'll leave you with this: we were 100% certain that the robots.txt…

Thanks Scott. Please advice how to create network blocks. I run a affiliate site and need to do this for my site too!

Re: Who exactly is crawling my site?

#39

Post by ScottWhigham »

csharp_gooru wrote: post #37
ScottWhigham wrote: post #14

I actually did this exact robots.txt for my site this summer. The net effect? A massive loss of secondary market traffic - enough that, after a month, we probably lost 10-15% in revenue. It surprised us. We rolled the old robots.txt back into place and bam - life went back to normal. You can question my pithy 300-character explanation all you want but I'll leave you with this: we were 100% certain that the robots.txt…

Thanks Scott. Please advice how to create network blocks. I run a affiliate site and need to do this for my site too!

It depends on your setup. We have a hardware firewall so I add/remove networks daily to it. If you are on apache, you can use htaccess. On a Windows server, you can add them to Windows Firewall.

Re: Who exactly is crawling my site?

#40

Post by RKearney »

bryanlarsen wrote: post #28

I've got a calendar site that's used by "GET /calls?month=2&year=7206 HTTP/1.1" 200 2423 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; + http://www.google.com/bot.html)" It wouldn't be hard to stop (404 nonsensical years & disable the previous / next links that will get them there), but it is amusing that Googlebot has clicked on that "next month" link over 60000 times...

I would think implementing something like this[0] would help.

[0] https://support.google.com/webmasters/answer/96569?hl=en

Post reply on HN