It is not possible to detect and block Chrome headless

Post by 0xdeadbeefbabe » Thu, Jan 18, 2018, 7:36 PM UTC

troels wrote: post #27

jimrandomh wrote: post #6

Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…

In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…

Post by lawl » Thu, Jan 18, 2018, 7:37 PM UTC

eric_h wrote: post #49

lawl wrote: post #47

Earlier quoted context omitted.

That only works if the scrapers are either in a country where you can do something about it. Also poisoning only works for a while. As soon as they detect the poisoning, they can easily figure out what tripped the scraping detection, and now you need to poison in an even more subtle way because the scrapers know what to look for. You can't win this game. Especially the "obvious" type of browser checks, where you can…

> Claiming you're on windows while actually running on a Linux VM, your TCP fingerprint gives it away This is interesting, I’d not heard of this approach. What’s the technique for generating such a fingerprint?

Post by adrr » Thu, Jan 18, 2018, 7:41 PM UTC

troels wrote: post #27

jimrandomh wrote: post #6

Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…

In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…

Post by patcheudor » Thu, Jan 18, 2018, 7:43 PM UTC

dsjoerg wrote: post #43

_m8fo wrote: post #26

Isn't it impossible to win the game of blocking headless browsers? What's stopping someone from creating an API that opens up a real browser, uses a real (or virtual) keyboard, types in/clicks the real address, etc. then proceeds to use computer vision to scrape the information from the page without touching the DOM?

You are in principle correct, but in practice you need to account for the side channels of information as well -- does the mouse and keyboard behave like a human or a robot? Are there thousands upon thousands of sessions coming from the same IP address? The cat and mouse game happens at every level, not just the DOM/browser-detection level.

Post by saas_co_de » Thu, Jan 18, 2018, 7:46 PM UTC

jjeaff wrote: post #15

merb wrote: post #9

if you want to block scrapers, just add rate limiting...

That's old hat and ineffective. Scrapers usually proxy through large lists of rotating ip addresses. There are lots of services for it.