Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It is not possible to detect and block Chrome headless

intoli.com

Post reply on HN
Print view View original

91–100 of 174 posts

Re: It is not possible to detect and block Chrome headless

#91

Post by j_coder »

tlrobinson wrote: post #88
j_coder wrote: post #87

It is "easy" to block scraping. Make it very costly to scrape: - Render your page using canvas and WebAssembly compiled from C, C++, or Rust. Create your own text rendering function. - Have multiple page layouts - Have multiple compiled versions of your code (change function names, introduce useless code, different implementations of the same function) so it is very difficult reverse engineer, fingerprint and patch.…

All of which is defeated by OCR.

Yes. The idea here is to make you dependent on OCR (you also have to find where is the information as the page design changes) and to waste a lot of your server resources making it very costly to scrape.

Re: It is not possible to detect and block Chrome headless

#92

Post by _o_ »

icebraining wrote: post #77
_o_ wrote: post #72

All those tests are useless and effective only against script kiddys (which are now like 99.99999% of developers by old standards) and are unable to code anything else but crappy languages like js. For people grown up with web, capable of coding in c/c++ those tests are a joke, I'll just modify the source code to return what is expected and 'game over'. We were reversing drms by dissasembling and patching the binarie…

What does C/C++ have to do with this, when the point of the article is showing that they can be defeated using JS?

JS is run within c/c++ js engine that can be modified to return you whatever fake results. You can't prevent that. As always, any lock is cheaper to defeat than create.

Re: It is not possible to detect and block Chrome headless

#93

Post by mykull »

jimrandomh wrote: post #6

Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…

it's an entirely pointless arms race, where the scraper always wins, because web browsers are just a piece of software saying the right things at the right time over TCP. The only deterrent to scraping is the cost of running a web browser to interact using a real javascript stack, so for some really lame spammy uses of scrapers (the most common) headless browsers are often not cost effective.

Re: It is not possible to detect and block Chrome headless

#94

Post by foob »

Sephr wrote: post #50

The author's navigator.webdriver fix is easily detected, though of course it is fixable with changes to Chrome. This cat and mouse game probably isn't worth pursuing against dedicated adversaries. if (navigator.webdriver || Object.getOwnPropertyDescriptor(navigator, 'webdriver')) { // navigator.webdriver exists or was redefined }

That test actually wouldn't work:

    > navigator.webdriver
    true
    > Object.getOwnPropertyDescriptor(navigator, 'webdriver')
    undefined
As you say though, it's a cat and mouse game and you could always override the behavior of getOwnPropertyDescriptor() if it were used in a test.

Re: It is not possible to detect and block Chrome headless

#95

Post by xfer »

_o_ wrote: post #79

This article is a joke, all those methods of "protections" are a joke. What we called "script kiddys" and are now a major amount of so called developers are just underdeveloped lamers who just don't know that the fight is lost in advance. All the methods that you take are useless when you get into situation of scraper run by someone who is able to modify (oh and is able to code in c/c++) and recompile the client side…

Harsh words; but very true that the developers don't realize the chromium is open-source.. maybe they should just jump to the new drm extension, atleast that will challenge the dedicated scrapers.

Re: It is not possible to detect and block Chrome headless

#96

Post by mykull »

_o_ wrote: post #79

This article is a joke, all those methods of "protections" are a joke. What we called "script kiddys" and are now a major amount of so called developers are just underdeveloped lamers who just don't know that the fight is lost in advance. All the methods that you take are useless when you get into situation of scraper run by someone who is able to modify (oh and is able to code in c/c++) and recompile the client side…

This is pretty broad criticism, but you are at least right that there is no sound principle on which to protect a website from "automated" interaction.

Re: It is not possible to detect and block Chrome headless

#97

Post by Retric »

dsacco wrote: post #75
troels wrote: post #27

Earlier quoted context omitted.

In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…

> When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Having written bespoke scraping systems professionally, I think you overestimate the applicability of this technique. For one thing, detecting that a scraper is a scraper is the problem, not the prelude to the problem. You might as well block them at that point, if you feel you can reliably dete…

Choosing a believable alteration is just as important as detecting a browser. For price scrapping even 3% changes to prices can do a lot of damage and is much harder to detect than completely bogus data.

Re: It is not possible to detect and block Chrome headless

#98

Post by eastendguy »

tlrobinson wrote: post #88
j_coder wrote: post #87

It is "easy" to block scraping. Make it very costly to scrape: - Render your page using canvas and WebAssembly compiled from C, C++, or Rust. Create your own text rendering function. - Have multiple page layouts - Have multiple compiled versions of your code (change function names, introduce useless code, different implementations of the same function) so it is very difficult reverse engineer, fingerprint and patch.…

All of which is defeated by OCR.

Good point. OCR powered web scraping is even available out of the box nowadays.

https://a9t9.com/kantu/docs/scraping#ocr

Re: It is not possible to detect and block Chrome headless

#99

Post by icebraining »

_o_ wrote: post #92
icebraining wrote: post #77

Earlier quoted context omitted.

What does C/C++ have to do with this, when the point of the article is showing that they can be defeated using JS?

JS is run within c/c++ js engine that can be modified to return you whatever fake results. You can't prevent that. As always, any lock is cheaper to defeat than create.

No, you misunderstood; you can defeat the detection using JS. You don't need C at all.

Re: It is not possible to detect and block Chrome headless

#100

Post by moyix »

jimrandomh wrote: post #6

Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…

A friend and I wrote a paper about techniques for doing this a while back (never formally published; it was just a class project). My favorite technique for avoiding automated scan environments was to show goatse.cx and then run your code in the onUnload handler when the human frantically tries to close the tab.

http://www.cs.columbia.edu/~brendan/honeymonkey.pdf

Post reply on HN