Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How are you dealing with scraping hits from EC2 machines?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#11

Post by rli »

adrianoconnor wrote: post #4

Do you have a robots.txt? That's the standard way.

robots.txt can be ignored, it's just a reference for honest spiders. I think the way described above, of listing top requestors, doing statistics and then automating blocking is indeed the best way. Could also be there's a blocklist or two around of malicious scrapers. And if there isn't, that's a new business proposal.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#14

Post by dumbfounder »

digitalpbk wrote: post #5
chewxy wrote: post #3

Let's examine your motivation: why do you want to block said scrapers in the first place? SEO concerns (dupe content)?

Mostly duplicate content & messing up my analytics (increased bounce rate, decreased time spend on page etc.)

What analytics are you using? I thought most would use javascript to avoid problems like this, and I would wager that the vast majority of bots don't bother executing javascript. You will always have legit bots hitting your site as well.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#15

Post by digitalpbk »

digitalpbk wrote: post #5
chewxy wrote: post #3

Let's examine your motivation: why do you want to block said scrapers in the first place? SEO concerns (dupe content)?

Mostly duplicate content & messing up my analytics (increased bounce rate, decreased time spend on page etc.)

Because it seems to be from selenium (from referer), it is triggering the JS too, we are using Google Analytics.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#18

Post by ig1 »

dumbfounder wrote: post #10
adrianoconnor wrote: post #4

Do you have a robots.txt? That's the standard way.

Calling them "scrapers" implies they are doing something nefarious (stealing content). Robots.txt is for law abiding bots.

Not really. "Scraping" just refers to extracting data from a site using an automated method, it doesn't have any connotations about the motivation or acceptability of the process.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#19

Post by ig1 »

digitalpbk wrote: post #15
digitalpbk wrote: post #5

Earlier quoted context omitted.

Mostly duplicate content & messing up my analytics (increased bounce rate, decreased time spend on page etc.)

Because it seems to be from selenium (from referer), it is triggering the JS too, we are using Google Analytics.

Why not exclude the EC2 ip range from analytics ?
Post reply on HN