Live data from Hacker News

It is not possible to detect and block Chrome headless

intoli.com

151–160 of 174 posts

Re: It is not possible to detect and block Chrome headless

#151
post #74

Earlier quoted context omitted.

If it's a video site, I expect the video to play when I land, e.g. youtube. I'm initiating on purpose by browsing

The first thing I do when hitting a YouTube URL is stop the video. Then I'll either run youtube-dl on the URL, or just paste it straight into a proper video player (VLC).

There are Chrome extensions that automatically pause the video.

Re: It is not possible to detect and block Chrome headless

#152
post #150
post #55

Earlier quoted context omitted.

Poisoning the well is very effective. We employed it at a large ecommerce company that was getting hit by carders testing credit cards on low price point items(sub $5). We were playing cat and mouse with them for six months. Found certain attributes about the browser that the botnet was using and fed them randomized success/fail responses. After two weeks of feeding them bad data, they left and never came back. They…

The whole cat and mouse game thing... for some strange reason that sounds fun to me. Probably because I don't know the details and the workload involved in actually doing it (and it's not my money or inventory at stake). It seems like it would be exciting, in that somewhat naive juvenile-ish fantasy sort of way, to try and figure out how to mitigate the threat, implement it quickly, and deploy it to watch it play out…

It is incredibly fun. I set up a Google Voice number specifically for site content managers to call me when comment/review spammers came around. I was fast enough at blocking spammers repeated attempts at evading registration blocking(email domains, IP address ranges, browser disguising) that I would usually make the spammers give up within an hour.

Re: It is not possible to detect and block Chrome headless

#153
post #144

Earlier quoted context omitted.

I dont stop crawlers, I only randomly feed damaged/wrong data to crawlers. I especially loving doing this for e-commerce sites. Now the table has turned. Try guess which fraction of your scrapped data were wrong.

My point is you’re not detecting the ones that you should be the most concerned with.

Problem is as developers, sometimes we only have hammers for these screws.

Re: It is not possible to detect and block Chrome headless

#154

Earlier quoted context omitted.

I wonder how long it will be before someone comes up with the idea of using iPhone style facial recognition to tell whether a human is looking at the TV/Monitor or not.

Oh please no, exposing those sorts of APIs will quickly be utilized by ad-tech guys to make interstitial video ads that don't go away until you finish watching them.

Black Mirror S01E02

Re: It is not possible to detect and block Chrome headless

#155
post #61

Earlier quoted context omitted.

And I can attest that they often presume that I'm a robot. At this point it would be easier for me to write an alternative frontend to Google search (or just use duckduckgo), but it was be amusing to think that I might evade this by writing a script to simulate mouse movements to appear less robotic.

> And I can attest that they often presume that I'm a robot. In my experience this occurs when you are either doing this too much, or you are not accepting their cookies when logged in. (I don't recall the behavior when logged out.)

For the record: I was logged out / incognito.

Actually, for whatever reason, this also often seems to cause reCapcha to essentially hellban me and just keep asking for me to solve an endless series of capchas. :/

Re: It is not possible to detect and block Chrome headless

#156
post #55
post #27

Earlier quoted context omitted.

In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…

Poisoning the well is very effective. We employed it at a large ecommerce company that was getting hit by carders testing credit cards on low price point items(sub $5). We were playing cat and mouse with them for six months. Found certain attributes about the browser that the botnet was using and fed them randomized success/fail responses. After two weeks of feeding them bad data, they left and never came back. They…

Your comment is why I feel automating engineers would be very hard. Cat and mouse game sounds like it requires a human... and a cat... and a mouse...

Re: It is not possible to detect and block Chrome headless

#157
post #150
post #55

Earlier quoted context omitted.

Poisoning the well is very effective. We employed it at a large ecommerce company that was getting hit by carders testing credit cards on low price point items(sub $5). We were playing cat and mouse with them for six months. Found certain attributes about the browser that the botnet was using and fed them randomized success/fail responses. After two weeks of feeding them bad data, they left and never came back. They…

The whole cat and mouse game thing... for some strange reason that sounds fun to me. Probably because I don't know the details and the workload involved in actually doing it (and it's not my money or inventory at stake). It seems like it would be exciting, in that somewhat naive juvenile-ish fantasy sort of way, to try and figure out how to mitigate the threat, implement it quickly, and deploy it to watch it play out…

There are aspects that are fun, but I feel like if you're doing it right, it is stressful. You are playing an antagonistic game with bad actors, so there's risk, and you'd better be well past just gaming out the probabilities and costs there. Just because you noticed them doesn't mean they can't do damage. You'd also better get informed buy-in from other relevant departments, etc.

Quite a while ago, I was involved in baiting an attacker in a somewhat different way, but with the same goal (destroying the value we were providing to them). After the attacker figured out what was going on, they issued a somewhat credible threat to damage the company's machines (they included some details demonstrating they had access to a couple internal machines at some point), attacked and DOSed them, and persistently tried spearphishing them for months afterwards.

I guess I'd just say, (a) doing this sort of thing responsibly sucks a lot of the fun out of it, and (b) don't underestimate the risks of things going pear-shaped. You could be buying yourself a lot of ongoing grief. Something as everyday as that spearphishing attack can be nerve wracking - even after making annoying the crap out of everyone by repeating how to be careful with email, there's no way to be sure it won't hit, and the next thing you know people are sitting around with upper management having conversations nobody wants to have about network segmentation and damage mitigation.

Re: It is not possible to detect and block Chrome headless

#158
post #17

The EME DRM is part of the game for those who really want to block headless. It will arrive, sooner or later.

I always assumed DRM would eventually factor into this. I’ve only ever read about it in the context of media, but I’m assuming there’s ways to use it creatively for fingerprinting and blocking scraping as well. Do you have any links with insights into that?

I am not familiar with the specific details and how they would allow this, but ... if it is for human consumption and not bot consumption, it is enough to render the result into a DRMd h264

Re: It is not possible to detect and block Chrome headless

#159
post #76

Earlier quoted context omitted.

I think this is a good example of "poisoning the well" in practice, and I was in a similar position as you describe when I was working in incident response at a consumer bank a few years ago. That said, this is a very particular scenario, and I don't think you can generalize the effectiveness of the technique from this example. In situations where attackers are looking for boolean responses, i.e. to verify email addr…

Man. I really like that idea, but is giving a random "success" to a supposed credit card charge PCI compliant?

PCI is largely misunderstood. Spoofing a success on a transaction has nothing to do with proper storage and transmission of credit card information.

Most e-commerce sites will use a payment processor such as Braintree or Stripe so they don't even have to deal with PCI.

PCI applies to entities who store and process their own payments, e.g. Target, Amazon.

Re: It is not possible to detect and block Chrome headless

#160
post #112

If you want to detect if a human is visiting your site, open an ad popup with a big close button directly over the content. A human being will always, 100% of the time, immediately close the popup. Automation won't care.

OK, but that is guaranteed to annoy users. Plus, I think you’re underestimating the intelligence of the people writing scrapers — obviously they’re going to visit the site manually and see what appears to be a fingerprinting measure. Then they’ll update the scraper to close that pop up. There are no effective solutions to this problem.

[deleted]
Post reply on HN