Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It is not possible to detect and block Chrome headless

intoli.com

Post reply on HN
Print view View original

61–70 of 174 posts

Re: It is not possible to detect and block Chrome headless

#61

Post by nicklaf »

roywiggins wrote: post #46
_m8fo wrote: post #26

Isn't it impossible to win the game of blocking headless browsers? What's stopping someone from creating an API that opens up a real browser, uses a real (or virtual) keyboard, types in/clicks the real address, etc. then proceeds to use computer vision to scrape the information from the page without touching the DOM?

Rate-limiting followed by CAPTCHAs seems to be the usual strategy. I think Google claims to try to detect humans by parsing out their mouse movements and scroll events.

And I can attest that they often presume that I'm a robot.

At this point it would be easier for me to write an alternative frontend to Google search (or just use duckduckgo), but it was be amusing to think that I might evade this by writing a script to simulate mouse movements to appear less robotic.

Re: It is not possible to detect and block Chrome headless

#62

Post by fixermark »

I'm not sure why one wants to bother to do this.

With tools like Sikuli script (sikuli.org) already around for ages, automating a headed browser isn't rocket science. So the best-case scenario for detecting headless browsers is "The bad guys just use headed browsers and another automation solution."

Re: It is not possible to detect and block Chrome headless

#64

Post by kbenson »

ThrustVectoring wrote: post #42
kbenson wrote: post #25

Earlier quoted context omitted.

All the passive techniques are much harder to reasons about, but much easier to match. You just look at the complete request/response headers, make sure you match them, and have some good sources to request from. Much harder is stuff like Distil's script injection, where they transparently inject script tags that do fingerprinting, and they obfuscate the code that does so annoyingly (it's not really hard to reverse,…

Did you do a cost estimate for the "Wizard of Oz" solution of having real people with real browsers (and a script to pull data from the site?) Might have been worthwhile.

It was actually two changes, one which required a much more intensive request regime because of a public caching (SOLR) system change, and then the much more aggressive scraping detection. The first caused us to change from requesting data 78 times an hour (items changed within 2 minutes requested every minute, 6 minutes of changes every 5 minutes, 11 minutes of changes every 10 minutes for overlapping coverage) to many thousands of checks an hour for much less accurate information. In the end, we ended up doing very targeted checks and much less accuracy for different classes of items. Having people actually do the checking just wouldn't be feasible for our size and resources (very small company, <10 employees), even through mechanical turk (I suspect).

Re: It is not possible to detect and block Chrome headless

#65

Post by goerz »

urlgrey wrote: post #35

Crawlers & scrapers that rely on headless browsers like Chrome often initiate playback of video on the pages they access. The company I work for (Mux) has a product that collects user-experience metrics for video playback in browsers & native apps. It's been a non-trivial effort developing a system to identify video views from headless browsers so that we might limit their impact on metrics. Being able to make this d…

Video should never play unless actively initiated by the user. That would fix the metrics, as the headless browser probably wouldn't initiate the video playback

Re: It is not possible to detect and block Chrome headless

#66

Post by tyingq »

jimrandomh wrote: post #6

Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…

"is an arms race that's been going on for a long time"

I'd say not always. Some websites attract a high number of unsophisticated scrapers, and a few clever ones. If it's high volume, blocking the former is often worth it just for the reduced load. I agree that going to war with skilled and funded scrapers is futile.

Re: It is not possible to detect and block Chrome headless

#67

Post by srirachahot »

_m8fo wrote: post #26

Isn't it impossible to win the game of blocking headless browsers? What's stopping someone from creating an API that opens up a real browser, uses a real (or virtual) keyboard, types in/clicks the real address, etc. then proceeds to use computer vision to scrape the information from the page without touching the DOM?

One simple reason is resource cost. Having a non headless browser is more expensive. Therefore at scale you are wasting resources.

Re: It is not possible to detect and block Chrome headless

#68

Post by averagewall »

nicklaf wrote: post #58
dsjoerg wrote: post #43

Earlier quoted context omitted.

You are in principle correct, but in practice you need to account for the side channels of information as well -- does the mouse and keyboard behave like a human or a robot? Are there thousands upon thousands of sessions coming from the same IP address? The cat and mouse game happens at every level, not just the DOM/browser-detection level.

So record actual user input data and generate similar input patterns stochastically. That said if you try to scale this up beyond what a reasonable, normal user world do in one sitting, you are bound to stand out. Although that said, I find that I trigger such rate-limiting mechanisms already as a human just when searching Google as a human being and clicking through every last search result page.

You'd have to scrape slowly to mimic a real slow user. Maybe at that point you'd be cheaper to get Mechanical Turk to do it. That should solve IP rate limiting, captchas, and just about everything except the endless arms race. Why are so many people going directly to these same-formatted internal URLs without clicking through from random other places? So the site can change the internal URLs and break it all over again.

Re: It is not possible to detect and block Chrome headless

#69

Post by adrianmonk »

srirachahot wrote: post #67
_m8fo wrote: post #26

Isn't it impossible to win the game of blocking headless browsers? What's stopping someone from creating an API that opens up a real browser, uses a real (or virtual) keyboard, types in/clicks the real address, etc. then proceeds to use computer vision to scrape the information from the page without touching the DOM?

One simple reason is resource cost. Having a non headless browser is more expensive. Therefore at scale you are wasting resources.

Yes, it's more expensive than a headless browser. Impossible to say, though, whether it's more expensive than can be justified in achieving an objective until that objective is known.

Re: It is not possible to detect and block Chrome headless

#70

Post by Savageman »

eric_h wrote: post #49
lawl wrote: post #47

Earlier quoted context omitted.

That only works if the scrapers are either in a country where you can do something about it. Also poisoning only works for a while. As soon as they detect the poisoning, they can easily figure out what tripped the scraping detection, and now you need to poison in an even more subtle way because the scrapers know what to look for. You can't win this game. Especially the "obvious" type of browser checks, where you can…

> Claiming you're on windows while actually running on a Linux VM, your TCP fingerprint gives it away This is interesting, I’d not heard of this approach. What’s the technique for generating such a fingerprint?

The most trivial check is using the TCP packet TTL (Time To Live). Windows and Linux doesn't have the same default value so it's "easy" to recognize which one sent a specific packet.

Just look up for "ttl windows vs linux" to find tons of article about it :)

Post reply on HN