Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It is not possible to detect and block Chrome headless

intoli.com

Post reply on HN
Print view View original

1–10 of 174 posts

Re: It is not possible to detect and block Chrome headless

#2

Post by heipei »

Interesting follow-up (again). It will be very interesting to see where attempts to detect headless browser will first appear in the wild. Once we know that and the prevalence, we can make a judgement call on how much effort to put into anti-detection techniques. It's an arms race for sure, but once you know your target you can evaluate whether you even have to put up the effort to defeat a non-existent adversary.

Re: It is not possible to detect and block Chrome headless

#5

Post by nukeop »

Good, the less effective various spying techniques are, and the easier they are to throw off, the better the internet is for its users. I don't want any website owners to know what device, browser, or other program, I use to access their site, and they have no business knowing that. I like it being a piece of information I can supply voluntarily for my own purposes, and I get the heebie jeebies every time I read about a new shady fingerprinting technique that exploits some new, previously unexplored quirk of web technologies.

Re: It is not possible to detect and block Chrome headless

#6

Post by jimrandomh »

Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level.

The main context in which Javascript tries to detect whether it's being run headless is when malware is trying to evade behavioral fingerprinting, by behaving nicely inside a scanning environment and badly inside real browsers. The main context in which a headless browser tries to make itself indistinguishable from a real user's web browser is when it's trying to stop malware from doing that. Scrapers can piggyback on the latter effort, but scraper-detectors can't really piggyback on the former. So this very strongly favors the scrapers.

Re: It is not possible to detect and block Chrome headless

#7

Post by zzzcpan »

"That’s when it becomes impossible. You can come up with whatever tests you want, but any dedicated web scraper can easily get around them."

As long as the logic is hidden from the scrapers, i.e. not running in a web browser, scrapers are at a disadvantage. They don't have the data about the users that websites have. And even something as simple as Accept-Language header associated with an IP subnet is a data point that can be used to protect against scraping. There are a lot more data points though and more aggressive fingerprinting can effectively destroy scraping.

Re: It is not possible to detect and block Chrome headless

#8

Post by zzzcpan »

nukeop wrote: post #5

Good, the less effective various spying techniques are, and the easier they are to throw off, the better the internet is for its users. I don't want any website owners to know what device, browser, or other program, I use to access their site, and they have no business knowing that. I like it being a piece of information I can supply voluntarily for my own purposes, and I get the heebie jeebies every time I read abou…

This incentivizes more aggressive fingerprinting, not the other way around. Too bad people don't realize it.
Post reply on HN