Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How are you dealing with scraping hits from EC2 machines?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#31

Post by brechin »

Scraping, in itself, is often not prohibited, wrong, illegal, or against most sites' TOS. Your site would allow me, for example, to scrape all your content for my own personal use, but I couldn't re-publish or re-sell the info.

It seems hard to limit legitimate uses of a free resource without changing the requirements on how users access the site (require account signup, use CAPTCHAs, use CSS/JS to only display properly in a browser).

As one who does a lot of scraping, I have encountered few barriers that can't be (legally) overcome with a reasonable amount of effort.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#32

Post by bdcravens »

bdcravens wrote: post #24
chewxy wrote: post #23

Earlier quoted context omitted.

So the solution is simple isn't it? Google Analytics filter > filter referers like Selenium (or the entire EC2 block).

Not sure how you could filter Selenium. A well written script looks like normal traffic.

Not all scrapers spider a site's contents. Sometimes they are going after something specific.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#33

Post by brechin »

seats wrote: post #30
bdcravens wrote: post #24

Earlier quoted context omitted.

Not sure how you could filter Selenium. A well written script looks like normal traffic.

A well written scraper looks like normal traffic in the same way that a well written pseudo random number generator looks like a random number generator. It'll fool your eye but not statistical analysis. Think about the goal of a scraper, it needs to actually walk through all the content. That doesn't look like a normal user at all. An individual request might look ok, but in aggregate the pattern of a robot pops out…

So, would that detection mechanism be able to deal with a number of coordinated scrapers rotating through lists of proxies, using different User-Agent strings, making requests with (pseudo-)random delays between requests?

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#34

Post by pbhjpbhj »

brechin wrote: post #31

Scraping, in itself, is often not prohibited, wrong, illegal, or against most sites' TOS. Your site would allow me, for example, to scrape all your content for my own personal use, but I couldn't re-publish or re-sell the info. It seems hard to limit legitimate uses of a free resource without changing the requirements on how users access the site (require account signup, use CAPTCHAs, use CSS/JS to only display prope…

>Your site would allow me, for example, to scrape all your content for my own personal use //

On what basis are you claiming this. Sounds like it would be true under fair-use clauses of US Copyright law but it's certainly not true in the UK (and by extension I presume for you to perform on content served from the UK though I've yet to read a thorough treatment of how the [ie any] law works with server locations).

Commercial considerations are usually much broader than selling too: not only could you not resell it but you couldn't distribute it (whether by publishing or otherwise).

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#36

Post by brechin »

pbhjpbhj wrote: post #34
brechin wrote: post #31

Scraping, in itself, is often not prohibited, wrong, illegal, or against most sites' TOS. Your site would allow me, for example, to scrape all your content for my own personal use, but I couldn't re-publish or re-sell the info. It seems hard to limit legitimate uses of a free resource without changing the requirements on how users access the site (require account signup, use CAPTCHAs, use CSS/JS to only display prope…

> Your site would allow me, for example, to scrape all your content for my own personal use // On what basis are you claiming this. Sounds like it would be true under fair-use clauses of US Copyright law but it's certainly not true in the UK (and by extension I presume for you to perform on content served from the UK though I've yet to read a thorough treatment of how the [ie any] law works with server locations). Co…

From TOS on his site:

"Cucumbertown authorizes you to view, download and/or print the Materials only for personal, non-commercial use, provided that you keep intact all copyright and other proprietary notices contained in the original Materials."

So, for example, I could grab everything from the site (minus copyrighted images, etc.) and make my own personal DB of the content. Obviously that's a lot of effort for a little reward for one person, but if I created a repo with a set of tools for people to do this for themselves it could become a big legitimate source of "scraping" traffic.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#37

Post by seats »

brechin wrote: post #33
seats wrote: post #30

Earlier quoted context omitted.

A well written scraper looks like normal traffic in the same way that a well written pseudo random number generator looks like a random number generator. It'll fool your eye but not statistical analysis. Think about the goal of a scraper, it needs to actually walk through all the content. That doesn't look like a normal user at all. An individual request might look ok, but in aggregate the pattern of a robot pops out…

So, would that detection mechanism be able to deal with a number of coordinated scrapers rotating through lists of proxies, using different User-Agent strings, making requests with (pseudo-)random delays between requests?

yup, read the paper

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#40

Post by lifeguard »

logn wrote: post #29
bdcravens wrote: post #25

I've done a lot of automation with Selenium, on EC2. (For the curious, it was on behalf of clients with legit access, not wholesale pillaging of a public resource) 1) You can block EC2 wholesale. You've mentioned issues with this, and can be bypassed via VPN or using another network. EC2 is attractive because it's so cheap (with spot instances, starts at 0.3 cents per hour), but it's not the only option. 2) Timing. N…

> 3) Report addresses to Amazon. I really don't know if they'd take action. They take action. I've seen it done.

They take action on abuse. I am not sure scraping is abuse.
Post reply on HN