Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who exactly is crawling my site?

danbirken.com

Post reply on HN
Print view View original

21–30 of 80 posts

Re: Who exactly is crawling my site?

#21

Post by cgore »

At the bottom of the page he mentions Yelp, and looking at their robots.txt they have this weird stuff for each of the allowed bots:

  Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-0
And over and over ten times. Trying to see if the bots ignore robots.txt I guess?

Re: Who exactly is crawling my site?

#22

Post by tmarthal »

elchief wrote: post #15

What about a system like this: You crawl your own site every day (or generate the same content into files) Put the content into files with file names representing the URLs (or HAR format?) Zip em up Put them on bittorrent Tell the search engines to look there for your content Wouldn't that save everyone a lot of work?

That was almost the original intent behind sitemap.xml, (http://sitemaps.org). Except of course, that they were being used erroneously and changes published when no content was changed, and so the spiders stopped using them as canon.

Re: Who exactly is crawling my site?

#23

Post by toomuchtodo »

ryandrake wrote: post #20
nostromo wrote: post #13

Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers. I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic. I feel like search engines sh…

Chicken and egg. Should they 1. crawl you first, or 2. send you traffic first? If 1, how long until they should either send traffic or stop, and if 2, how?

Could they not run webhooks where my site can call to them when to do a full or partial crawl? PubSub for web crawling, if you will.

Re: Who exactly is crawling my site?

#24

Post by bediger4000 »

devicenull wrote: post #7
bediger4000 wrote: post #6

So, you have a problem with Ahrefs, too? But what about Ezooms? That's a very annoying bot for my site, drives zero traffic. Also, what about Cyveillance? They show up with a lying User-Agent field ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)"), but run Linux, and they ignore robots.txt by not even asking for it. I actually blocked Ahrefs, Ezooms, Yandex and Cyveillance by IP address range in httpd.conf for a…

Does Ahrefs not respect robots.txt? Their website says they do, https://ahrefs.com/robot/index.php

I don't know if I tried that. Originally, I wanted to let them know they were forbidden, but now I just want to jerk them around.

Re: Who exactly is crawling my site?

#25

Post by ceejayoz »

toomuchtodo wrote: post #23
ryandrake wrote: post #20

Earlier quoted context omitted.

Chicken and egg. Should they 1. crawl you first, or 2. send you traffic first? If 1, how long until they should either send traffic or stop, and if 2, how?

Could they not run webhooks where my site can call to them when to do a full or partial crawl? PubSub for web crawling, if you will.

Sitemaps (http://www.sitemaps.org/) help a bit in that regard, as crawlers can check the sitemap and only crawl updated content.

Re: Who exactly is crawling my site?

#26

Post by ceejayoz »

cgore wrote: post #21

At the bottom of the page he mentions Yelp, and looking at their robots.txt they have this weird stuff for each of the allowed bots: Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-0 And over and over ten times. Trying to see if the bots ignore robots.txt I guess?

Probably a honeypot, yeah.

Re: Who exactly is crawling my site?

#27

Post by pavel_lishin »

ryandrake wrote: post #20
nostromo wrote: post #13

Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers. I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic. I feel like search engines sh…

Chicken and egg. Should they 1. crawl you first, or 2. send you traffic first? If 1, how long until they should either send traffic or stop, and if 2, how?

Why not adjust the crawl rate based on how often the site's content changes? If DuckDuckGoogle crawls me once an hour for a week, and notices that I only updated my content twice, why not scale the crawling back to a more reasonable rate, like once a day?

Re: Who exactly is crawling my site?

#28

Post by bryanlarsen »

I've got a calendar site that's used by "GET /calls?month=2&year=7206 HTTP/1.1" 200 2423 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

It wouldn't be hard to stop (404 nonsensical years & disable the previous / next links that will get them there), but it is amusing that Googlebot has clicked on that "next month" link over 60000 times...

Re: Who exactly is crawling my site?

#29

Post by TheLoneWolfling »

birken wrote: post #19
TheLoneWolfling wrote: post #2

"I've decided to block all crawlers to the site other than Google or Bing" And this is why I find people that respond to privacy complaints about Google with "if you want to switch search engines no-one is stopping you" frustrating. Additionally, I would like to point out that according to those numbers, there are ~41,000 (418,814 - 199,725 - 40,359 - 36,340 - 33,893 - 26,325 - 13,458 - 10,657 - 6,109 - 5,993 - 4,959…

This is a fair point, however blacklisting isn't necessarily a perfect solution either. It would require continuous manual effort in going through the logs and blocking bad bots, and if some new bot were to misbehave and crawl too aggressively, blacklisting would only help after the fact. I do think I did make a mistake though. To your point, I shouldn't block crawlers that both behave and are attempting to help my s…

Whitelisting a couple of bots now doesn't help at all for any new search engines trying to start up. What are they to do, contact every site admin individually?

> if some new bot were to misbehave and crawl too aggressively, blacklisting would only help after the fact.

In that case, don't blacklist all bots, simply add a crawl delay to any bots that you haven't specifically allowed:

User-agent: * Crawl-delay: 10

This allows minor bots to continue to crawl the site, while cutting back on bandwidth costs for the couple of ones that are being overly aggressive.

Re: Who exactly is crawling my site?

#30

Post by yogo »

elchief wrote: post #15

What about a system like this: You crawl your own site every day (or generate the same content into files) Put the content into files with file names representing the URLs (or HAR format?) Zip em up Put them on bittorrent Tell the search engines to look there for your content Wouldn't that save everyone a lot of work?

This would be another dream come true for SEO. Too easy to game.
Post reply on HN