Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What do people use to prevent crawlers?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 123 posts

Re: Ask HN: What do people use to prevent crawlers?

#101

Post by bad_user »

inglor wrote: post #19
bad_user wrote: post #9

The other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaScript protections the website has. You can also run such scripts with a headless browser for full automation, PhantomJS being an obvious choice. You really can't protect against this unless you start making the experience of regular visitors much worse.

Sure you can protect against this - there are several companies that use machine learning to spot small differences between selenium and real users (mouse delays etc). For example, it might detect that a mouse click is dispatched at exact intervals (and block it). To which you'd think "I'll just add Math.random() * 2000" which it'll easily detect as well. It's _definitely_ doable, but it's not as trivial as recording…

The issue here isn't that it can't be done, the problem being one of false positives.

By doing this you must accept that a certain percentage of legitimitate users, which can be quite significant, will be blocked. In case you're wondering, yes, this does happen with solutions such as Cloudflare.

And at that point, either your website isn't popular, in which case you can't afford to lose users, or it is very popular, in which case you'll piss of enough users as to receive bad reviews.

Basically you can't afford to do this, unless you're Facebook or Google, and you have to then wonder why Facebook or Google do not deploy such protections

So going back to my main point, of course it's possible, but the experience for users gets significantly worse such that you won't want to do it.

Re: Ask HN: What do people use to prevent crawlers?

#102

Post by monodeldiablo »

calafrax wrote: post #61
monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

That is good advice from a technical standpoint but from a legal standpoint creating security features that prevent scraping gives you a clearer cause of action against scrapers so if someone starts making a lot of money off your content you get leverage to force them to pay for it.

You can achieve the same thing, from a legal perspective, with a well-placed statement of IP ownership.

Re: Ask HN: What do people use to prevent crawlers?

#103

Post by monodeldiablo »

jorgec wrote: post #46
monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

You are comparing two different things. To have a free website is not the same than to unlimited grants such as a guy to taking the whole pie. Or worst, lets say that you writes a free essay and everybody could read it and share. However, somebody takes it, deletes your name and put his name instead.

Then you, my friend, have a copyright issue. There are lots of legal tools for addressing this, and they're all cheaper than wasting your energy trying to counter the bots.

If your free content is the sole source of your online revenue, then your reputation is your business. Nefarious crawlers who rebrand your content cannot, by definition, beat you to market. So you stand to gain much more by writing great content, driving readers to it as soon as it's posted, and filing the occasional DMCA takedown than trying to compete with plagiarists in a game of whack-a-mole.

Re: Ask HN: What do people use to prevent crawlers?

#104

Post by inglor »

bad_user wrote: post #101
inglor wrote: post #19

Earlier quoted context omitted.

Sure you can protect against this - there are several companies that use machine learning to spot small differences between selenium and real users (mouse delays etc). For example, it might detect that a mouse click is dispatched at exact intervals (and block it). To which you'd think "I'll just add Math.random() * 2000" which it'll easily detect as well. It's _definitely_ doable, but it's not as trivial as recording…

The issue here isn't that it can't be done, the problem being one of false positives. By doing this you must accept that a certain percentage of legitimitate users, which can be quite significant, will be blocked. In case you're wondering, yes, this does happen with solutions such as Cloudflare. And at that point, either your website isn't popular, in which case you can't afford to lose users, or it is very popular,…

I've found Cloudflare's solution to be very inadequate compared to Distil or Incapsula.

Both Distil and Incapsula _did_ have a small number of false positives (showing a captcha to users). We did have to write some code to overcome that.

Re: Ask HN: What do people use to prevent crawlers?

#105

Post by rk06 »

alecco wrote: post #81
wrath wrote: post #22

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…

Did you respect robots.txt?

Considering the effort that went into it.

I am pretty sure crawling robot.txt links was their P1 requirement.

Re: Ask HN: What do people use to prevent crawlers?

#106

Post by winkv »

bad_user wrote: post #9

The other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaScript protections the website has. You can also run such scripts with a headless browser for full automation, PhantomJS being an obvious choice. You really can't protect against this unless you start making the experience of regular visitors much worse.

Interesting, i am facing problem with this approach.There is a site which pops up alert box for username and password and i can't seem to switch to it with chrome web extension.With selenium it is as simple as driver.switchTo().alert(); have you ever faced this problem? by any chance you know how to fix it? I am trying to accomplish the same thing as described in this question in SO https://sqa.stackexchange.com/questions/20710/not-able-to-us...

Re: Ask HN: What do people use to prevent crawlers?

#107

Post by jklein11 »

blowski wrote: post #50
monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

I run a property website that lists properties for sale, rent, etc. A big part of my job is importing feeds, scraping sites (with permission) - and preventing others scraping our site. I know that some people will scrape, but I make I try to make it unprofitable for them to do so. We do a bunch of other stuff, like adding fake properties so we can check who is scraping our content, and using tarpits. Developers alway…

If your secret sauce is the data that you have why not sell it to your competitor?

Re: Ask HN: What do people use to prevent crawlers?

#108

Post by wrath »

DrScump wrote: post #99
wrath wrote: post #22

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…

If unwanted scraping can be distinguished from legitimate traffic, wouldn't a sort of honeypot strategy work such that you then provide those requests you've identified as likely to be unwelcome with fake or divergent data?

When websites get a ton of traffic the concern is that the algorithm that find fake data will not be accurate and start blocking paying customers. So it's a fine line between blocking paying customers and fake data. What these algorithms do instead of blocking is to throw captcha's so if the traffic is really human, the captcha can be solved. The bigger problem is that there's a good chance that humans who are thrown a captcha will leave to go buy somewhere else (either because they are lazy, the captcha is hard, etc...).

Solutions like CloudFare and Distill have sophisticated algorithm to balance out fake and real data but even they are not close to being perfect.

Re: Ask HN: What do people use to prevent crawlers?

#109

Post by laughfactory »

ne0free wrote: post #69
wrath wrote: post #22

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…

How do you bypass google recaptcha

I do it using rotating proxies, stripping cookies between requests, randomly varying the delay between requests, randomly selecting a valid user-agent string, etc. It's a pain in the butt. And to scrape more than I do, faster than I do, would be pretty freaking expensive in terms of time and money.

Note that Google is pretty aggressive about captcha-ing "suspicious" activity and/or throttling responses to suspicious requests. You can easily trigger a captcha with your own manual searching. Just search for something, go to page 10, and repeat maybe 5-20 times and you'll see a captcha challenge.

If Google gets more serious about blocking me then I'll use ML to overcoming their ML (which should be doable because they're always worried about keeping Search consumer-friendly).

Re: Ask HN: What do people use to prevent crawlers?

#110

Post by laughfactory »

blowski wrote: post #49
huffmsa wrote: post #34

Earlier quoted context omitted.

Side bar, if only Twitter would realize this and turn the firehose back on and charge $x per month for API access. They'd be profitable in a month.

Given the number of times people assert this, I think Twitter must have explored this option and decided it wasn't a viable option. I don't know why, but I'll assume that they're smart enough to have decided against it for a good reason.

I would not make the assumption. I've been working for corporate America long enough to know that there's an astonishing level of incompetence many places. I'm sure Twitter is no exception. I'm sure _someone_ at Twitter has considered this, but perhaps they work for someone who doesn't "get it." Among many other possibilities.
Post reply on HN