Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Detecting Chrome headless, the game goes on

antoinevastel.com

Post reply on HN
Print view View original

11–20 of 143 posts

Re: Detecting Chrome headless, the game goes on

#11

Post by shawnz »

mcescalante wrote: post #10
itake wrote: post #2

This would be more interesting if the author explained this technique. People that are knowledgeable enough will deep dive into the webpage, but for everyone else, expect disappointment.

I'm not sure if diving deep into the page will yield results of how it's done. The page's javascript does a POST to a backend with the browser's fingerprint, and the server does all the "magic" where we can't see it. Unless there is new fingerprint info that is being sent to the server that wasn't around before, I'm skeptical about the javascript in the page revealing the full technique.

He claims the fingerprint library's techniques aren't used for the check though, so surely there must be an observable difference between the POST request from headless and non-headless

Edit: According to other commenters there are checks in the included version of the library which are not in the release version.

Re: Detecting Chrome headless, the game goes on

#12

Post by nfRfqX5n »

mcescalante wrote: post #10
itake wrote: post #2

This would be more interesting if the author explained this technique. People that are knowledgeable enough will deep dive into the webpage, but for everyone else, expect disappointment.

I'm not sure if diving deep into the page will yield results of how it's done. The page's javascript does a POST to a backend with the browser's fingerprint, and the server does all the "magic" where we can't see it. Unless there is new fingerprint info that is being sent to the server that wasn't around before, I'm skeptical about the javascript in the page revealing the full technique.

only way to do it these days... although the payload is not hashed or obfuscated in any way so it would be extremely easy to fake if it's even being stored in a db or memory somewhere, else you can just copy the request exactly as is

Re: Detecting Chrome headless, the game goes on

#13

Post by lol768 »

There are additional tests included in https://arh.antoinevastel.com/javascripts/fpCollect.min.js that do not exist in the GitHub repository over at https://github.com/antoinevastel/fp-collect.

  redPill: function() {
      for (var e = performance.now(), n = 0, t = 0, r = [], o = performance.now(); o - e

Re: Detecting Chrome headless, the game goes on

#14

Post by CaliforniaKarl »

I’ve participated in a number of Stanford research studies, and what the author is doing here is similar to part of it.

The studies in which I’ve participated always start with a statement of what they are generally looking for in a participant. You then take a survey that confirms if you are qualified. You then given a release to sign (and keep a copy of), which states what you’ll be doing, and providing an IRB contact. You then go through the study.

At the end of your participation, you are asked “What do you think the study is about?”, and then you were told the real purpose of the study. Eventually the paper(s) is/are published, with hypothesis, methodology, and results.

This seems similar: You decide if you want to participate, and are participating; the only thing that’s missing is the final paper.

Re: Detecting Chrome headless, the game goes on

#15

Post by zxcvbn4038 »

itake wrote: post #2

This would be more interesting if the author explained this technique. People that are knowledgeable enough will deep dive into the webpage, but for everyone else, expect disappointment.

Agreed that the article is poorly written for the Hacker News crowd, would be nice to have a description of his technique so the merits/faults could be analyzed without everyone conducting a reverse engineering effort.

It’s sad to have a smart guy like this dedicating his academic career to something this inconsequential. Anyone with enough incentive is going to be able to defeat any technique this guy dreams up. Anyone who might benefit on paper from detecting a headless browser isn’t going to want this because any possibility of a false positive is a missed impression, or a missed sales opportunity, or an ADA lawsuit (US), or an angry customer.

Re: Detecting Chrome headless, the game goes on

#16

Post by Pinbenterjamin »

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers.

Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless Chrome and selenium to build out most of our integrations, I'm starting to wonder if the science of blocking scraping is getting more popular...

I don't think what I'm doing is subversive at all, we're running background checks on people, and we can reduce business costs by eliminating error-prone researchers with smart scrapers that run all day.

I don't want to seem like the bad guy here, but what if I wanted to do the opposite of this research? Where do I start? Study the chromium source? Can anyone recommend a few papers?

Re: Detecting Chrome headless, the game goes on

#17

Post by jadell »

lol768 wrote: post #13

There are additional tests included in https://arh.antoinevastel.com/javascripts/fpCollect.min.js that do not exist in the GitHub repository over at https://github.com/antoinevastel/fp-collect . redPill: function() { for (var e = performance.now(), n = 0, t = 0, r = [], o = performance.now(); o - e

It doesn't seem to be using the Javascript. Looking at the page source, it has already made the determination before the Javascript runs.

If I load the page source in Chrome, it already includes the "You are not Chrome headless" message, but when I run it in a scraper I maintain, the page source loads with the "You are Chrome headless" message, even without running any Javascript.

Re: Detecting Chrome headless, the game goes on

#18

Post by Buttons840 »

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

Have you tried running your browsers in virtual frame buffers? Do they still get detected?

Re: Detecting Chrome headless, the game goes on

#19

Post by turtlebits »

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

The right thing to do would be to reach out to those sites and see if they is they have paid options for getting the data you need.

Re: Detecting Chrome headless, the game goes on

#20

Post by jadell »

mcescalante wrote: post #10
itake wrote: post #2

This would be more interesting if the author explained this technique. People that are knowledgeable enough will deep dive into the webpage, but for everyone else, expect disappointment.

I'm not sure if diving deep into the page will yield results of how it's done. The page's javascript does a POST to a backend with the browser's fingerprint, and the server does all the "magic" where we can't see it. Unless there is new fingerprint info that is being sent to the server that wasn't around before, I'm skeptical about the javascript in the page revealing the full technique.

The "You are/are not" message seems to be included in the page source before any Javascript runs. Is it possible there are detectable differences in the original HTTP request itself?
Post reply on HN