Live data from Hacker News

Ask HN: How are you dealing with scraping hits from EC2 machines?

news.ycombinator.com

41–42 of 42 posts

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#41
Many have suggested editing your robots.txt. This is absolutely the first step you should take. You could try blocking the crawlers by name or limit the request rate with a crawl delay in the robots.txt.

If the crawler ignores your robots.txt, check it's name in your access logs. Often, people build things and set them loose without thinking about the consequences. Many crawlers have a homepage / programmer contact information somewhere on the web. Let them know they are hammering your website.

What is the rate at which requests are being made? Are they making 1000 requests per second? Downloading tons of images? You should probably just ignore it if it is less than 1 request per second.

Re: Ask HN: How are you dealing with scraping hits from EC2 machines?

#42
You could add a CAPTCHA to suspect client IPs? You could implement some sort of real-time log analysis / monitoring based on client IP, x-forward-for (?) headers and User Agent, also possibly block certain geo-IP ranges from non-core customers. Inserting a javascript "puzzle" is also a way to test that a browser is generating the traffic and not a script.
Post reply on HN