Live data from Hacker News

Ask HN: What do people use to prevent crawlers?

news.ycombinator.com

121–123 of 123 posts

Re: Ask HN: What do people use to prevent crawlers?

#121
post #69

Earlier quoted context omitted.

How do you bypass google recaptcha

I do it using rotating proxies, stripping cookies between requests, randomly varying the delay between requests, randomly selecting a valid user-agent string, etc. It's a pain in the butt. And to scrape more than I do, faster than I do, would be pretty freaking expensive in terms of time and money. Note that Google is pretty aggressive about captcha-ing "suspicious" activity and/or throttling responses to suspicious…

If you do go the ML route, I recommend TensorFlow + Google Cloud (Both for the cost performance, and the irony).

Re: Ask HN: What do people use to prevent crawlers?

#122

Earlier quoted context omitted.

That is good advice from a technical standpoint but from a legal standpoint creating security features that prevent scraping gives you a clearer cause of action against scrapers so if someone starts making a lot of money off your content you get leverage to force them to pay for it.

You can achieve the same thing, from a legal perspective, with a well-placed statement of IP ownership.

It is not that open-and-shut.

Unauthorized access to a computer system is a different legal category than IP licensing.

Post reply on HN