Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It is not possible to detect and block Chrome headless

intoli.com

Post reply on HN
Print view View original

81–90 of 174 posts

Re: It is not possible to detect and block Chrome headless

#81

Post by fixermark »

Dolores12 wrote: post #80
fixermark wrote: post #62

I'm not sure why one wants to bother to do this. With tools like Sikuli script (sikuli.org) already around for ages, automating a headed browser isn't rocket science. So the best-case scenario for detecting headless browsers is "The bad guys just use headed browsers and another automation solution."

Looks like great tool, never heard of sikuli before. Thanks for the tip!

I name-drop it every chance I get ;) We used it to automate the integration tests for a game engine at a previous company; worked great, because it allowed us to fire events into the engine itself based upon the actual rendered pixels (Sikuli supports varying levels of fuzzy image detection for event targets).

Re: It is not possible to detect and block Chrome headless

#82

Post by RickHull »

AugustJok wrote: post #52
hfern wrote: post #39

Earlier quoted context omitted.

Mailinator does the same thing -- http://mailinator.blogspot.com/2011/05/how-to-get-gmailcom-b... I don't see anything wrong with it.

I don't see anything wrong with it, I am just impressed the approach.

Evil is an interesting choice of words to mean not wrong and impressive...

Re: It is not possible to detect and block Chrome headless

#83

Post by imiric »

adrr wrote: post #55
troels wrote: post #27

Earlier quoted context omitted.

In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…

Poisoning the well is very effective. We employed it at a large ecommerce company that was getting hit by carders testing credit cards on low price point items(sub $5). We were playing cat and mouse with them for six months. Found certain attributes about the browser that the botnet was using and fed them randomized success/fail responses. After two weeks of feeding them bad data, they left and never came back. They…

It's interesting that it came down to actual virtual warfare.

Good on you for sticking through it.

Re: It is not possible to detect and block Chrome headless

#85

Post by mehrdadn »

nicklaf wrote: post #61
roywiggins wrote: post #46

Earlier quoted context omitted.

Rate-limiting followed by CAPTCHAs seems to be the usual strategy. I think Google claims to try to detect humans by parsing out their mouse movements and scroll events.

And I can attest that they often presume that I'm a robot. At this point it would be easier for me to write an alternative frontend to Google search (or just use duckduckgo), but it was be amusing to think that I might evade this by writing a script to simulate mouse movements to appear less robotic.

> And I can attest that they often presume that I'm a robot.

In my experience this occurs when you are either doing this too much, or you are not accepting their cookies when logged in. (I don't recall the behavior when logged out.)

Re: It is not possible to detect and block Chrome headless

#86

Post by Johnny555 »

troels wrote: post #27
jimrandomh wrote: post #6

Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…

In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…

Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would.

If you detect wrongly though, you risk alienating your users -- if I saw obviously incorrect prices on an ecommerce site, I wouldn't shop there, assuming that they are incompetent and if I enter any payment information it's going to be whisked away by hackers.

Depending on your industry etc., it may be viable to take the legal route. If you suspect who the scraper is, you can deliberately plant a 'trap street', then look for it at your suspect. If it shows up, let lose the lawyers.

Depending on your industry etc., it may be viable to take the legal route. If you suspect who the scraper is, you can deliberately plant a 'trap street', then look for it at your suspect. If it shows up, let lose the lawyers

That may be harder than you suspect, given that it's not illegal to violate a website's terms of use:

https://www.eff.org/document/oracle-v-rimini-ninth-circuit-o...

Even if you have an enforceable contract, you can still face an expensive legal battle if they choose to fight (and it's likely not their first fight), so hopefully you have some deep pockets.

Re: It is not possible to detect and block Chrome headless

#87

Post by j_coder »

It is "easy" to block scraping. Make it very costly to scrape:

- Render your page using canvas and WebAssembly compiled from C, C++, or Rust. Create your own text rendering function.

- Have multiple page layouts

- Have multiple compiled versions of your code (change function names, introduce useless code, different implementations of the same function) so it is very difficult reverse engineer, fingerprint and patch.

- Try to prevent debugging by monitoring time interval between function calls, compare local time interval with server time interval to detect sandboxes.

- Always encrypt data from server using different encryption mechanisms every time.

- Hide the decryption key into random locations of your code (use generated multiple versions of the code that gets the key)

- Create huge objects in memory and consume a lot of CPU (you may mine some crypto coins) for a brief period of time (10s) on the first visit of the user. Make very expensive for the scrapers to run the servers. Save an encrypted cookie to avoid doing it later. Monitor concurrent requests from the same cookie.

The answer is that it is possible but it will cost you a lot.

Re: It is not possible to detect and block Chrome headless

#88

Post by tlrobinson »

j_coder wrote: post #87

It is "easy" to block scraping. Make it very costly to scrape: - Render your page using canvas and WebAssembly compiled from C, C++, or Rust. Create your own text rendering function. - Have multiple page layouts - Have multiple compiled versions of your code (change function names, introduce useless code, different implementations of the same function) so it is very difficult reverse engineer, fingerprint and patch.…

All of which is defeated by OCR.

Re: It is not possible to detect and block Chrome headless

#89

Post by nybble41 »

bpicolo wrote: post #74
goerz wrote: post #65

Earlier quoted context omitted.

Video should never play unless actively initiated by the user. That would fix the metrics, as the headless browser probably wouldn't initiate the video playback

If it's a video site, I expect the video to play when I land, e.g. youtube. I'm initiating on purpose by browsing

Regarding YouTube in particular, I tend to open up videos in background tabs for later viewing and find it very annoying that they start playing automatically before I get around to that tab. I did go there to watch the video—eventually. Just not the second that the page finishes loading.

YMMV. A persistent setting to enable or disable auto-play would be ideal.

Re: It is not possible to detect and block Chrome headless

#90

Post by OkGoDoIt »

beagle3 wrote: post #17

The EME DRM is part of the game for those who really want to block headless. It will arrive, sooner or later.

I always assumed DRM would eventually factor into this. I’ve only ever read about it in the context of media, but I’m assuming there’s ways to use it creatively for fingerprinting and blocking scraping as well. Do you have any links with insights into that?
Post reply on HN