Ask HN: How are you dealing with scraping hits from EC2 machines?

#1

Post by digitalpbk » Wed, Dec 12, 2012, 2:00 PM UTC

My website is being constantly hit by scrapers from EC2 machines http://d.pr/i/bLtE . I went aggressive and blocked all access from Ec2 IP’s http://bit.ly/SUOaof until I realized that quite a few reader proxies like Flipboard are based out of Ec2 and blanket blocking of these amazon machines wont help. How is the community dealing with this problem? Can you advice?

Edit: I’ve seen somewhere that Stackoverflow blocks all the ec2 machines. I don’t think this is the most optimal solution considering many legit services. Also the hits come from different ip’s.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#2

Post by gingerjoos » Wed, Dec 12, 2012, 2:08 PM UTC

So the referer is a local Selenium server? How did you figure out it was an EC2 machine?

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#3

Post by chewxy » Wed, Dec 12, 2012, 2:10 PM UTC

Let's examine your motivation: why do you want to block said scrapers in the first place? SEO concerns (dupe content)?

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#4

Post by adrianoconnor » Wed, Dec 12, 2012, 2:13 PM UTC

Do you have a robots.txt? That's the standard way.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#5

Post by digitalpbk » Wed, Dec 12, 2012, 2:17 PM UTC

chewxy wrote: post #3

Let's examine your motivation: why do you want to block said scrapers in the first place? SEO concerns (dupe content)?

Mostly duplicate content & messing up my analytics (increased bounce rate, decreased time spend on page etc.)

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#6

Post by dumbfounder » Wed, Dec 12, 2012, 2:19 PM UTC

I had problems with people scraping Twicsy so hard that it was taking the site down. For a while I would manually review the top IP addresses requesting pages a couple times per day and look for patterns and ban IP's based on that. Then I created a script based on the patterns I recognized to do it automatically.

But then I just made Twicsy fast enough to deal with the traffic so I don't need to worry about it anymore. I guess it depends on your business model whether or not that will work for you.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#7

Post by dumbfounder » Wed, Dec 12, 2012, 2:21 PM UTC

gingerjoos wrote: post #2

So the referer is a local Selenium server? How did you figure out it was an EC2 machine?

That second link digitalpbk gives shows all public EC2 IP address ranges: http://bit.ly/SUOaof

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#8

Post by covati » Wed, Dec 12, 2012, 2:23 PM UTC

dumbfounder wrote: post #6

I had problems with people scraping Twicsy so hard that it was taking the site down. For a while I would manually review the top IP addresses requesting pages a couple times per day and look for patterns and ban IP's based on that. Then I created a script based on the patterns I recognized to do it automatically. But then I just made Twicsy fast enough to deal with the traffic so I don't need to worry about it anymor…

We've blocked a few of the worst, but mostly just added servers to deal with the load.

We actually found out who one of the worst ones was and contact them. It turns out it was a major legit proxy, but they had a bug in their proxy code that caused refetching of one of our urls over and over. They were very easy to work with and they fixed the bug.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#9

Post by digitalpbk » Wed, Dec 12, 2012, 2:24 PM UTC

gingerjoos wrote: post #2

So the referer is a local Selenium server? How did you figure out it was an EC2 machine?

Most of the IPs are from the EC2 subnet.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#10

Post by dumbfounder » Wed, Dec 12, 2012, 2:24 PM UTC

adrianoconnor wrote: post #4

Do you have a robots.txt? That's the standard way.

Calling them "scrapers" implies they are doing something nefarious (stealing content). Robots.txt is for law abiding bots.