Ask HN: What do people use to prevent crawlers?

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will.

What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of pages, but there was a whole team dedicated to keeping it going). If you have money to spend, there's Distill Networks or Incapsula that have good solutions. They block PhantomJS and browsers that use Selenium to navigate websites, as well as rate limit the bots.

What I found really affective that some websites do is tarpit bots. That is, slowly increase the number of seconds it takes to return the http request. So after a certain amount of request to your site it takes 30+ seconds for the bot to get the HTML back. The downside is that your web servers need to accept many more incoming connections but the benefit is you'll throttle the bots to an acceptable level.

I currently run a website that gets crawled a lot, deadheat.ca. I've written a simple algorithm that tarpits bots. I also throw a captcha every now and then when I see an IP address hits too often over a span of a few minutes. The website is not super popular and, in my case, it's pretty simple to differentiate between a human or bot.

Hope this helps...

If you're getting a lot of crawler traffic, your site probably has information a lot of people find useful, so you should consider finding a way to monetize it.

Otherwise, your best bet (hardest to get around in my experience) is monitoring for actual user I/O. Like if someone starts typing in an input field, real humans have to click on it beforehand, and most bots won't.

Or if a user clicks next-page without the selector being visible or without scrolling the page at all. Not natural behavior.

Think like a human.

My favorite thing was to identify bots and instead of blocking them, switch to a slightly scrambled data set to make the scrape useless but look good to the developer who stole it. It was a ton of fun as a side project. I'd also suggest you add some innocent fake data to your real site and then set up google alerts of all of the above to catch traffic. About 50% of sites would respond positively to an email when you showed them they were hosting fake data. About 90% would take my data down if that was followed up with a stronger C&D. One key is to catch them fast, while they're still a little nervous about showing off their stolen data online.

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?"

I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss interrupted.

Him: "WTF are you doing?"

Me: "We're trying to figure out how to prevent people from scraping our data."

Him: "WTF do you want to do that for?"

Me: "Uh... to prevent them from stealing our data."

Him: "But we put it on the public Web..."

Me: "Yeah, but that data took thousands of compute hours to grind out. They're getting a valuable product for free!"

Him: "So then pull it from Web."

Me: "But then we won't get any sales from people who see that we published this new and exciting-- Oh. I see what you mean."

Him: "Yeah, just get a list of the top 20 IP addresses, figure out who's scraping, and hand it off to our sales guys. Scraping ain't free, and our prices aren't high. This is a sales tool, and it's working. Now get back to building shit to make our customers lives easier, not shittier."

Sure enough, most of the scrapers chose to pay rather than babysit web crawlers once we pointed out that our price was lower than their time cost. If your data is valuable enough to scrape, it's valuable enough to sell.

The only technological way to prevent someone crawling your website is to not put it on a publicly-facing property in the first place. If you're concerned about DoS or bandwidth charges, throttle all users. Otherwise, any attempts to restrict bots is just pissing into the wind, IMHO.

Spend your energies on generating real value. Don't engage in an arms racw you're destined to lose.

Exactly. If you have data so valuable that people want to take it, it's going to be a lot easier to figure out how to sell it to them (they're probably not professional web scrapers, it's just a means to an end.) Than to waste yours, theirs and everyone else's time going tit-for-tat keeping them away from your data.