Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What do people use to prevent crawlers?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 123 posts

Re: Ask HN: What do people use to prevent crawlers?

#31

Post by lazyjones »

- permanently block Tor exit nodes and relays (some relays are hidden exit nodes)

- permanently block known anonymizer service IP addresses

- permanently block known server IP address ranges, such as AWS

- temporarily (short intervals, 5-15 mins) block IP addresses with typical scraping access patterns (more than 1-2 hits/sec over 30+ secs)

- add captchas

All of these will cost you a small fraction of legitimate users and are only worth it if scraping puts a strain on your server or kills your business model...

Re: Ask HN: What do people use to prevent crawlers?

#32

Post by StavrosK »

pbhjpbhj wrote: post #23
inglor wrote: post #19

Earlier quoted context omitted.

Sure you can protect against this - there are several companies that use machine learning to spot small differences between selenium and real users (mouse delays etc). For example, it might detect that a mouse click is dispatched at exact intervals (and block it). To which you'd think "I'll just add Math.random() * 2000" which it'll easily detect as well. It's _definitely_ doable, but it's not as trivial as recording…

IF you throttle the connection then wouldn't download time and network latency, etc., give you a better random?

You can generally tell because of the distribution, which tends to show up after lots of data points.

Re: Ask HN: What do people use to prevent crawlers?

#33

Post by lazyjones »

monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

> Sure enough, most of the scrapers chose to pay rather than babysit web crawlers once we pointed out that our price was lower than their time cost. If your data is valuable enough to scrape, it's valuable enough to sell.

Cool story, bro.

Some scrapers will buy your data if it's good enough, but most can't be identified (good luck with those Tor/AWS/dynamic IP users) and some will just resell your data at a lower price. So as a general strategy against scraping, this is useless.

Re: Ask HN: What do people use to prevent crawlers?

#34

Post by huffmsa »

monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

Side bar, if only Twitter would realize this and turn the firehose back on and charge $x per month for API access.

They'd be profitable in a month.

Re: Ask HN: What do people use to prevent crawlers?

#35

Post by huffmsa »

lazyjones wrote: post #33
monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

> Sure enough, most of the scrapers chose to pay rather than babysit web crawlers once we pointed out that our price was lower than their time cost. If your data is valuable enough to scrape, it's valuable enough to sell. Cool story, bro. Some scrapers will buy your data if it's good enough, but most can't be identified (good luck with those Tor/AWS/dynamic IP users) and some will just resell your data at a lower pri…

People with subscriptions are going to GIVE your data away for free.

Ever heard of KAZAA? Napster? BitTorrent?

All sites where people who have usually purchased data make it freely available to others.

Re: Ask HN: What do people use to prevent crawlers?

#36

Post by danpalmer »

huffmsa wrote: post #34
monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

Side bar, if only Twitter would realize this and turn the firehose back on and charge $x per month for API access. They'd be profitable in a month.

Twitter do charge for the firehose, and I hear it’s a reasonable amount of revenue. A person can’t go and buy it on a credit card, but there are authorised sellers, enterprise sales people, etc.

Bear in mind that it’s a large technical feat to be able to ingest the firehose effectively, so it’s not really suitable for consumers.

Re: Ask HN: What do people use to prevent crawlers?

#37

Post by huffmsa »

danpalmer wrote: post #36
huffmsa wrote: post #34

Earlier quoted context omitted.

Side bar, if only Twitter would realize this and turn the firehose back on and charge $x per month for API access. They'd be profitable in a month.

Twitter do charge for the firehose, and I hear it’s a reasonable amount of revenue. A person can’t go and buy it on a credit card, but there are authorised sellers, enterprise sales people, etc. Bear in mind that it’s a large technical feat to be able to ingest the firehose effectively, so it’s not really suitable for consumers.

I mean sell it piecemeal. Add a butt ton of endpoints. Make it easy to use, easy to integrate. Sell it on credit card.

Re: Ask HN: What do people use to prevent crawlers?

#38

Post by timbowhite »

I wrote a plugin for node.js/express that performs basic bot detection and bans bots by IP address until they pay you some Bitcoin:

project: https://github.com/timbowhite/botbouncer

simple demo: http://botbouncer.xyz/

I ran it for awhile on some medium traffic websites that were being heavily scraped. It blocked thousands of IP addresses, but IIRC only received one Bitcoin payment.

Re: Ask HN: What do people use to prevent crawlers?

#39

Post by digitalzombie »

You can use cloudfare but it's a small roadblock. I can still crawl that.

Also you can do frontend rendering, it's a bit larger roadblock but you can use phantomJS or something to crawl that.

IIRC there is a php framework that mutate your front end code but I'm not sure if it does it enough to stop a generalized xpath...

Also I used to work for company where they employ people full time for crawling. It will even notified the company if crawler stopped working so they can update their crawler...

Re: Ask HN: What do people use to prevent crawlers?

#40

Post by mattm »

One thing I've thought about but never had the chance to put into practice would be to randomize CSS classes and IDs. Most web scraping relies on these to identify the content they are looking for.

Imagine if everyday they changed? It would make things a lot more difficult.

There would be disadvantages to actual users with this method like caching wouldn't work very well but maybe this alternative site could be displayed only to bots.

The crawler could get smart about it and only use xpaths like the 6th div on the page so maybe in the daily update you could throw in some random useless empty divs and spans in various locations.

It's a lot of work to setup but I think you would make scraping almost impossible.

Post reply on HN