It is not possible to detect and block Chrome headless

Post by gildas » Thu, Jan 18, 2018, 6:51 PM UTC

sigotirandolas wrote: post #31

gildas wrote: post #16

The problem is that you can easily detect that some properties have been overloaded. For example, you can execute Object.getOwnPropertyDescriptor(navigator, "languages") to detect if navigator.languages is a native property or not.

It's possible to hide that as well, funnily enough, by also overwriting Object.getOwnPropertyDescriptor (and similar tricks). As far as I know, it's theoretically possible to use this trick to completely 'sandbox' some code so that there's no way it can detect certain functions being overwritten (by overwriting all functions such as Object.getOwnPropertyDescriptor, Function.toString, etc. and making them hide the ove…

Post by [deleted] » Thu, Jan 18, 2018, 6:59 PM UTC

thaumaturgy wrote: post #21

lr4444lr wrote: post #11

Earlier quoted context omitted.

I'll eat you through a proxy network then, unless you want to slow down your legitimate users too.

I now work for a company that is gathering metadata on the IP address space (in an effort to reduce the amount of abuse that sites and service providers have to deal with). It won't be very long before it'll be possible to identify most of the common proxying networks and block those. Scrapers can respond by setting up something like an ssh tunnel from a residential high speed connection to a remote server (so that s…

Post by tehlike » Thu, Jan 18, 2018, 7:03 PM UTC

Jach wrote: post #29

thaumaturgy wrote: post #21

Earlier quoted context omitted.

I now work for a company that is gathering metadata on the IP address space (in an effort to reduce the amount of abuse that sites and service providers have to deal with). It won't be very long before it'll be possible to identify most of the common proxying networks and block those. Scrapers can respond by setting up something like an ssh tunnel from a residential high speed connection to a remote server (so that s…

Maybe companies can instead focus on improving network infrastructure and software architecture to be able to eat any amount of scraper traffic since at some point it must become indistinguishable from a ddos that also needs to be handled...

Post by j_s » Thu, Jan 18, 2018, 7:04 PM UTC

Post by jrochkind1 » Thu, Jan 18, 2018, 7:10 PM UTC

troels wrote: post #27

jimrandomh wrote: post #6

Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…

In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…