Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who exactly is crawling my site?

danbirken.com

Post reply on HN
Print view View original

51–60 of 80 posts

Re: Who exactly is crawling my site?

#51

Post by seiyak »

"It turns out I'm not alone in adding these types of restrictions. Yelp blocks everybody but Google, Bing, ia_archiver (archive.org), ScoutJet (Blekko) and Yandex. LinkedIn also has a similar opt-in robots.txt, though they have whitelisted a larger number of bots than yelp."

At least we can contact/email Yelp and LinkedIn regarding to the crawlers if one can crawl or not according to their robots.txt. It's more generous than just allowing the big search engines such as Google and Bing. I'm not quite sure what's actually happening if we ask them though. I'll try that.

Re: Who exactly is crawling my site?

#52

Post by gizmo686 »

lostlogin wrote: post #48
pavel_lishin wrote: post #27

Earlier quoted context omitted.

Why not adjust the crawl rate based on how often the site's content changes? If DuckDuckGoogle crawls me once an hour for a week, and notices that I only updated my content twice, why not scale the crawling back to a more reasonable rate, like once a day?

This seems logical, however what about the person who updates rarely (once or twice a month) but get a lot of notice when they do. Search engines wouldn't want to miss the opportunity to get that traffic. I wonder if there is some sort of reverse process you could opt into - ie never crawl me until I ask to be crawled. Disclaimer: I know little about this sort of thing.

You could probably do this by having a robots.txt file that blocks all crawling. When you want to be re-crawled, you can edit it to allow the relevant spiders to crawl you. I would imagine that spiders do not automatically re-check sites which disallow everything often, but Google (and probably others) let you manually submit URLs to be crawled.

Re: Who exactly is crawling my site?

#53

Post by guard-of-terra »

nostromo wrote: post #13

Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers. I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic. I feel like search engines sh…

Do you/did you receive traffic from DDG? Because DDG uses Yandex for [some] organic search results and if you block its spider you will also partially block DDG.

Re: Who exactly is crawling my site?

#54

Post by dripton »

throwaway420 wrote: post #17
TheLoneWolfling wrote: post #2

"I've decided to block all crawlers to the site other than Google or Bing" And this is why I find people that respond to privacy complaints about Google with "if you want to switch search engines no-one is stopping you" frustrating. Additionally, I would like to point out that according to those numbers, there are ~41,000 (418,814 - 199,725 - 40,359 - 36,340 - 33,893 - 26,325 - 13,458 - 10,657 - 6,109 - 5,993 - 4,959…

> But don't ban everything just because a few (proximic and ADmantX) are hammering the site. I can understand blocking somebody that has a long-term and clear pattern of disrupting your site, not following the robots.txt rules, and not providing any links or anything back to you. But I find the idea of somebody preemptively blocking everything but Google and maybe Bing extremely distasteful. If everybody out there bl…

Or maybe all the other crawlers would just claim to be Googlebot. Just like all the other browsers (partially) claim to be Netscape.

Re: Who exactly is crawling my site?

#55

Post by csharp_gooru »

ScottWhigham wrote: post #39
csharp_gooru wrote: post #37

Earlier quoted context omitted.

Thanks Scott. Please advice how to create network blocks. I run a affiliate site and need to do this for my site too!

It depends on your setup. We have a hardware firewall so I add/remove networks daily to it. If you are on apache, you can use htaccess. On a Windows server, you can add them to Windows Firewall.

Cool. Thank you for your response. My site runs on a Windows server. I'll look into your recommendation. Thanks again.

Re: Who exactly is crawling my site?

#56

Post by FiloSottile »

I would really advise against disallowing the archive.org bot. Their mission is a really important one in the long run that would be disrupted by everyone adopting a whitelist approach and cutting them out, and I'm sure that their load is negligible.

Re: Who exactly is crawling my site?

#57

Post by nostromo »

guard-of-terra wrote: post #53
nostromo wrote: post #13

Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers. I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic. I feel like search engines sh…

Do you/did you receive traffic from DDG? Because DDG uses Yandex for [some] organic search results and if you block its spider you will also partially block DDG.

I did not know that! Thanks, I'll look.

Re: Who exactly is crawling my site?

#58

Post by PaulHoule »

nostromo wrote: post #13

Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers. I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic. I feel like search engines sh…

It's actually a lot worse if your site is large since crawlers will make even more requests crawling a large site.

If I see that a crawler is sending me any traffic at all, I will accept that, but if the amount of traffic is zero, I put them in robots.txt and in the IP block list. Although I try hard to make sure my images are all clean creative common images, I block the robots that are there to find copyrighted images because these ones exist to give me nothing but trouble.

Re: Who exactly is crawling my site?

#59

Post by walshemj »

gizmo686 wrote: post #52
lostlogin wrote: post #48

Earlier quoted context omitted.

This seems logical, however what about the person who updates rarely (once or twice a month) but get a lot of notice when they do. Search engines wouldn't want to miss the opportunity to get that traffic. I wonder if there is some sort of reverse process you could opt into - ie never crawl me until I ask to be crawled. Disclaimer: I know little about this sort of thing.

You could probably do this by having a robots.txt file that blocks all crawling. When you want to be re-crawled, you can edit it to allow the relevant spiders to crawl you. I would imagine that spiders do not automatically re-check sites which disallow everything often, but Google (and probably others) let you manually submit URLs to be crawled.

But a disallow all robots.txt will kill your sites traffic - its not going to work like you think it is.

Re: Who exactly is crawling my site?

#60

Post by pavel_lishin »

lostlogin wrote: post #48
pavel_lishin wrote: post #27

Earlier quoted context omitted.

Why not adjust the crawl rate based on how often the site's content changes? If DuckDuckGoogle crawls me once an hour for a week, and notices that I only updated my content twice, why not scale the crawling back to a more reasonable rate, like once a day?

This seems logical, however what about the person who updates rarely (once or twice a month) but get a lot of notice when they do. Search engines wouldn't want to miss the opportunity to get that traffic. I wonder if there is some sort of reverse process you could opt into - ie never crawl me until I ask to be crawled. Disclaimer: I know little about this sort of thing.

Presumably you could set a minimum interval - an hour, or a day. This would make sure that big updates are noticed, without putting undue stress on the server.
Post reply on HN