Live data from Hacker News

It is not possible to detect and block Chrome headless

intoli.com

21–30 of 174 posts

Re: It is not possible to detect and block Chrome headless

#21
post #9

if you want to block scrapers, just add rate limiting...

I'll eat you through a proxy network then, unless you want to slow down your legitimate users too.

I now work for a company that is gathering metadata on the IP address space (in an effort to reduce the amount of abuse that sites and service providers have to deal with).

It won't be very long before it'll be possible to identify most of the common proxying networks and block those.

Scrapers can respond by setting up something like an ssh tunnel from a residential high speed connection to a remote server (so that scraping run on the remote server appears to come from the residential connection) and using that as a proxy, but it adds another level of effort.

(I am generally on the scrapers' side on this and think it's ultimately futile to try to block all scrapers. Website administrators need to accept that anything they put on the internet is public and get over it.)

Re: It is not possible to detect and block Chrome headless

#22
post #16

The problem is that you can easily detect that some properties have been overloaded. For example, you can execute Object.getOwnPropertyDescriptor(navigator, "languages") to detect if navigator.languages is a native property or not.

Can’t I accomplish the same thing by compiling a modified version of headless chrome?

Yes I think it's possible but it's much harder to do, I guess.

Re: It is not possible to detect and block Chrome headless

#23
post #7

"That’s when it becomes impossible. You can come up with whatever tests you want, but any dedicated web scraper can easily get around them." As long as the logic is hidden from the scrapers, i.e. not running in a web browser, scrapers are at a disadvantage. They don't have the data about the users that websites have. And even something as simple as Accept-Language header associated with an IP subnet is a data point t…

If I can navigate to it using a normal Chrome instance, then a headless instance is going to have all of the same information (accept-language) as far as the server can detect. Chrome headless is Chrome and makes network calls in the exact same way. That's why client side headless detection even exists.

So the only way in which what you suggest would work is if the system is so aggressive that it actually blocks a normal Chrome instance, which is more hostile that most systems are. But this allows a user to change settings until they do get a correct response back, and then just have the headless browser use those settings.

Re: It is not possible to detect and block Chrome headless

#24

It is impossible to make headless and normal browser send 100% indistinguishable traffic. The timing of the browser requests is influenced by rendering that for the two versions will be always different.

It's not impossible; you could e.g. profile a real client's timing and introduce delays into the headless version. It's not zero work, but it's very much not impossible if you're sufficiently motivated.

Especially recently with e.g. https://hackaday.com/2018/01/06/lowering-javascript-timer-re... , high-precision timers in JS might not be available for all clients for reasons other than ~"they're headless and trying to scrape my site".

Re: It is not possible to detect and block Chrome headless

#25
post #7

"That’s when it becomes impossible. You can come up with whatever tests you want, but any dedicated web scraper can easily get around them." As long as the logic is hidden from the scrapers, i.e. not running in a web browser, scrapers are at a disadvantage. They don't have the data about the users that websites have. And even something as simple as Accept-Language header associated with an IP subnet is a data point t…

All the passive techniques are much harder to reasons about, but much easier to match. You just look at the complete request/response headers, make sure you match them, and have some good sources to request from.

Much harder is stuff like Distil's script injection, where they transparently inject script tags that do fingerprinting, and they obfuscate the code that does so annoyingly (it's not really hard to reverse, just time consuming and and annoying). They pair this with being a bit more user friendly by redirecting you to a CAPTCHA page if your fingerprinting hits some threshold which if you answer redirects you to the page you wanted, so users experience and inconvenience if there's a false positive, but still get access to what they wanted.

I was able to get around most of the passive fairly easily with Perl and LWP, and even the active stuff and CAPTCHA redirects (cookie_jar all serialized to DB so I could store request and represent it to a user to answer), but once they started tweaking their fingerprint script ever couple months/weeks that's when the equation shifted. Distil, as a solutions provider, gets to amortize their changes across all their customers, while I would have to spend the time de-obfuscating it. They could just assign a person to change it once a week and they would effectively halve my time to get any real work done, so without a collective effort of some sorts to combat them, I saw the writing on the wall. :/

The sad thing is that when we moved to API access, their APIs are hampered to the degree that it actually takes two orders of magnitude more requests each minute for a fraction of the accuracy (I was able to query changes over the last couple minutes previously, and now I have to query the entire item set of a subset of all containers, when there are tens of thousands of containers). :/ Lose lose, since our use case isn't even the main reason the site wanted to block scrapers.

Re: It is not possible to detect and block Chrome headless

#26
Isn't it impossible to win the game of blocking headless browsers?

What's stopping someone from creating an API that opens up a real browser, uses a real (or virtual) keyboard, types in/clicks the real address, etc. then proceeds to use computer vision to scrape the information from the page without touching the DOM?

Re: It is not possible to detect and block Chrome headless

#27

Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…

In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would.

Depending on your industry etc., it may be viable to take the legal route. If you suspect who the scraper is, you can deliberately plant a 'trap street', then look for it at your suspect. If it shows up, let lose the lawyers.

Of course, the very best solution is to not care about being scraped. If your problem is the load that is caused to your site, provide an api instead and make it easily discoverable.

Re: It is not possible to detect and block Chrome headless

#28
post #5

Good, the less effective various spying techniques are, and the easier they are to throw off, the better the internet is for its users. I don't want any website owners to know what device, browser, or other program, I use to access their site, and they have no business knowing that. I like it being a piece of information I can supply voluntarily for my own purposes, and I get the heebie jeebies every time I read abou…

The conclusion that this makes spying techniques more difficult is not at ALL what this article is saying. It is just saying that no fingerprinting is going to be able to distinguish between headless and not headless, but that is because there are too MANY variations, not because the variations are hard to detect. Nothing in this article gives any instruction or guidance on how to prevent your browser from being fingerprinted and tracked.

Re: It is not possible to detect and block Chrome headless

#29

Earlier quoted context omitted.

I'll eat you through a proxy network then, unless you want to slow down your legitimate users too.

I now work for a company that is gathering metadata on the IP address space (in an effort to reduce the amount of abuse that sites and service providers have to deal with). It won't be very long before it'll be possible to identify most of the common proxying networks and block those. Scrapers can respond by setting up something like an ssh tunnel from a residential high speed connection to a remote server (so that s…

Maybe companies can instead focus on improving network infrastructure and software architecture to be able to eat any amount of scraper traffic since at some point it must become indistinguishable from a ddos that also needs to be handled...

Re: It is not possible to detect and block Chrome headless

#30
post #26

Isn't it impossible to win the game of blocking headless browsers? What's stopping someone from creating an API that opens up a real browser, uses a real (or virtual) keyboard, types in/clicks the real address, etc. then proceeds to use computer vision to scrape the information from the page without touching the DOM?

There's a similar "analog hole" for video DRM, too.
Post reply on HN