Who exactly is crawling my site? • Hacker News Board

Re: Who exactly is crawling my site?

#41

Post by agwa » Fri, Oct 18, 2013, 8:25 PM UTC

I have found this set of htaccess rules to be extremely useful in blocking bad crawling behavior:

https://github.com/bluedragonz/bad-bot-blocker

Re: Who exactly is crawling my site?

#42

Post by ressaid1 » Fri, Oct 18, 2013, 8:35 PM UTC

What about the bots that are not identifying themselves as such?

Re: Who exactly is crawling my site?

#43

Post by jzs » Fri, Oct 18, 2013, 8:37 PM UTC

I'm glad to see that you have revised your list and added DuckDuckGo. It would be sad to see genuine search engines like duckduckgo be left out.

Re: Who exactly is crawling my site?

#44

Post by powertower » Fri, Oct 18, 2013, 8:41 PM UTC

That website has only 1 post. And the post is talking about "SEO" and the various search-engine names... In relation to a SEO business.

I kind of see what he is really trying to do...

Get Google to rank it from the start on long tails of "SEO + other-keyword(s)".

(*I'm not complaining, it's pretty smart to start out like that, and make it on HN's front-page)

Re: Who exactly is crawling my site?

#45

Post by Scryptonite » Fri, Oct 18, 2013, 9:02 PM UTC

I deal(t) with the same thing. I made it so my web server would try to stream a page that never ends, and some bots would stay connected for hours and hours. But over time they seem to have adapted.

I've also noticed some of them used to make requests synchronously (waiting for the previous to finish before making another), but they have adapted to make requests in parallel and add timeouts so they don't have their time wasted quite as long.

I created a log of the ones who stayed connected the longest.

https://gist.github.com/scryptonite/5324724

I don't bother to maintain it anymore, but it was pretty interesting watching them change tactics over time.

Re: Who exactly is crawling my site?

#46

Post by morley » Fri, Oct 18, 2013, 9:11 PM UTC

powertower wrote: post #44

That website has only 1 post. And the post is talking about "SEO" and the various search-engine names... In relation to a SEO business. I kind of see what he is really trying to do... Get Google to rank it from the start on long tails of "SEO + other-keyword(s)". (*I'm not complaining, it's pretty smart to start out like that, and make it on HN's front-page)

Somewhat strange that HN links don't have the rel="nofollow" attribute, or else this particular angle wouldn't work.

Re: Who exactly is crawling my site?

#47

Post by droidlabour » Fri, Oct 18, 2013, 9:11 PM UTC

What if a bot uses user-agent to look like it came from an end-user browser?

Re: Who exactly is crawling my site?

#48

Post by lostlogin » Fri, Oct 18, 2013, 9:12 PM UTC

pavel_lishin wrote: post #27

ryandrake wrote: post #20

Earlier quoted context omitted.

Chicken and egg. Should they 1. crawl you first, or 2. send you traffic first? If 1, how long until they should either send traffic or stop, and if 2, how?

Why not adjust the crawl rate based on how often the site's content changes? If DuckDuckGoogle crawls me once an hour for a week, and notices that I only updated my content twice, why not scale the crawling back to a more reasonable rate, like once a day?

This seems logical, however what about the person who updates rarely (once or twice a month) but get a lot of notice when they do. Search engines wouldn't want to miss the opportunity to get that traffic. I wonder if there is some sort of reverse process you could opt into - ie never crawl me until I ask to be crawled. Disclaimer: I know little about this sort of thing.

Re: Who exactly is crawling my site?

#49

Post by powertower » Fri, Oct 18, 2013, 9:18 PM UTC

morley wrote: post #46

powertower wrote: post #44

That website has only 1 post. And the post is talking about "SEO" and the various search-engine names... In relation to a SEO business. I kind of see what he is really trying to do... Get Google to rank it from the start on long tails of "SEO + other-keyword(s)". (*I'm not complaining, it's pretty smart to start out like that, and make it on HN's front-page)

Somewhat strange that HN links don't have the rel="nofollow" attribute, or else this particular angle wouldn't work.

From my understanding, due to massive abuse of the rel="nofollow" linkout, for the last several years Google has 1) ignored them by either still passing link-juice and/or incoming weight on terms (SEO), and 2) penalizes sites that use those types of links exclusively.

Re: Who exactly is crawling my site?

#50

Post by unhappyhippie » Fri, Oct 18, 2013, 9:21 PM UTC

ceejayoz wrote: post #25

toomuchtodo wrote: post #23

Earlier quoted context omitted.

Could they not run webhooks where my site can call to them when to do a full or partial crawl? PubSub for web crawling, if you will.

Sitemaps ( http://www.sitemaps.org/ ) help a bit in that regard, as crawlers can check the sitemap and only crawl updated content.

Thanks.