Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It is not possible to detect and block Chrome headless

intoli.com

Post reply on HN
Print view View original

161–170 of 174 posts

Re: It is not possible to detect and block Chrome headless

#161

Post by linkregister »

philipodonnell wrote: post #117
shock wrote: post #115

Earlier quoted context omitted.

if you intentionally provided false information to an otherwise legitimate user, and that user is harmed by your false information, isn't that a breach of contract that would make you liable for damages based on the harm your bad data caused? What contract?

Terms of service. It is not considered a contract if you are just an anonymous user, but once you take an affirmative action like creating an account you're explicitly agreeing to that and they are legally binding. The parent comment did not specify anonymous and plenty of bots create accounts to scrape with. But even aside from the explicit contractual terms, even without one, you cannot just run around acting in ba…

Since when do anonymous users not have to adhere to publicly-stated terms of service?

Re: It is not possible to detect and block Chrome headless

#162

Post by emmelaich »

It's a very dangerous thing to do for SEO reasons too.

I'm sure Google and others have automated user-like crawling which attempts to validate their official Google indexing bot.

If the results between the two differ in certain ways you may well get your site buried way down in search results.

Re: It is not possible to detect and block Chrome headless

#163

Post by wiz21c »

Could someone tell me why everybody wants to fight against headless browser ? If I want to use such a browser to browse your site, site that you voluntarily show to the public, then it's my problem, my code, not yours. If you want to protect your data so much, then maybe you shouldn't put them on the web first place. (yep, I present things in black and white, but you get the picture)

I would also add this :

https://www.bitlaw.com/copyright/database.html#Feist

because it basically says it's hard/pointless to protect data.

Re: It is not possible to detect and block Chrome headless

#164

Post by chopin »

fny wrote: post #120
jimrandomh wrote: post #6

Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…

Give it time my friend: they don't need to detect headless browsers, they just need to restrict your DOM-given freedom altogether. Some prick is bound to make some fancy non-DOM web framework using web assembly and turn the internet into a DRM-ridden mess.

Is there something like a non-DOM UI framework? Either you paint your pixels yourself or you have something equivalent to a DOM.

Even without fancy HTML5 stuff anybody could have sent a prerendered image.

I remember web-sites which exclusively rendered in Flash. I avoided them like a plague.

Re: It is not possible to detect and block Chrome headless

#165

Post by exikyut »

dsacco wrote: post #140
exikyut wrote: post #134

Earlier quoted context omitted.

> If you want to productively block scrapers, you should consider using a captcha-based system at the application layer, preferably a captcha that hasn't been broken yet and which can't be outsourced to a mechanical turk-based API. If nothing else, doing that will introduce at least 10 - 20 seconds of latency per request, which might be intolerable for many scrapers even if they're quite sophisticated. I'm thinking o…

> Firstly, I'm aware of human-based captcha-defeating systems. You describe captchas that "cannot be outsourced to a mechanical turk-based API". I'm wasn't aware such systems existed, that sounds scary. Do you have any examples? Google's latest captcha specification (and similarly sophisticated systems) must be completed in a small window of time, change rapidly, have a varying number of "rounds", and are extremely a…

> Google's latest captcha specification (and similarly sophisticated systems) must be completed in a small window of time, change rapidly, have a varying number of "rounds", and are extremely antagonistic to being reloaded in e.g. a frame.

Hmmm.

I've observed the timing thing myself - I'll sometimes hit the captcha first while I fill the rest of a form out, then it'll expire, and I'll need to do it again.

What do you mean by "change rapidly" and "varying number of "rounds""?

Regarding avoiding loading the content in a frame - what about making the initial iframe load happen on the machine of the person filling the captcha? Why can't this work?

--

> Practically speaking, you can't consistently outsource that to a third party API that uses humans to click and verify the images.

I'm both very curious and somewhat surprised this is the case.

--

>> It sounds plausible that one could simply sign up for the captcha-filling service multiple times, and submit captchas to defeat in parallel. Could this work?

> Sure, but what are you gaining? ... you'll be paying the same as just sending multiple requests to the API, and the concurrent requests are routed via the API's backend to different human operators for a solution.

This is what I'd strongly expect, but I very much wonder if captcha-filling services ratelimit the number of filled captchas per account.

In theory (now I think about it) ratelimiting would hurt the service provider so this might be unlikely, but where I want to play with massive parallelism I do wonder if sharding hundreds of requests across multiple accounts would be useful or not. Sharding across multiple providers would probably increase parallel throughput by a small amount though.

Thanks for the feedback!

Re: It is not possible to detect and block Chrome headless

#166

Post by embleton »

dsacco wrote: post #137
bsder wrote: post #135

Earlier quoted context omitted.

> As an example, let's say a certain company exposes sequentially crawlable online orders without authentication. Wow. That's starting to get really shady. Note to self: All things facing the web in ecommerce must be cryptographically randomized. :(

> Wow. That's starting to get really shady. Yes, there is a massive amount of crawling that happens in the service of financial forecasting. Satellite imagery, drones, web pages, API endpoints, receipt data (free financial aggregators sell this), location data (free geolocation services sell this), purchase history (free email clients sell this), etc. This rabbit hole goes very deep. Some of it is actively sold by fr…

For someone just getting into web development, would you point me to some resources that describe how to find and pull this data?

Re: It is not possible to detect and block Chrome headless

#167

Post by systoll »

anonytrary wrote: post #156
adrr wrote: post #55

Earlier quoted context omitted.

Poisoning the well is very effective. We employed it at a large ecommerce company that was getting hit by carders testing credit cards on low price point items(sub $5). We were playing cat and mouse with them for six months. Found certain attributes about the browser that the botnet was using and fed them randomized success/fail responses. After two weeks of feeding them bad data, they left and never came back. They…

Your comment is why I feel automating engineers would be very hard. Cat and mouse game sounds like it requires a human... and a cat... and a mouse...

GANs are an automated ‘cat and mouse game’

https://en.m.wikipedia.org/wiki/Generative_adversarial_netwo...

Re: It is not possible to detect and block Chrome headless

#168

Post by fny »

chopin wrote: post #164
fny wrote: post #120

Earlier quoted context omitted.

Give it time my friend: they don't need to detect headless browsers, they just need to restrict your DOM-given freedom altogether. Some prick is bound to make some fancy non-DOM web framework using web assembly and turn the internet into a DRM-ridden mess.

Is there something like a non-DOM UI framework? Either you paint your pixels yourself or you have something equivalent to a DOM. Even without fancy HTML5 stuff anybody could have sent a prerendered image. I remember web-sites which exclusively rendered in Flash. I avoided them like a plague.

A while ago Flipboard released a canvas based UI. https://github.com/Flipboard/react-canvas

Most companies do use some flash nonsense, but given the state of that, I'm sure they'll find some new insanity to foist on us.

Re: It is not possible to detect and block Chrome headless

#169

Post by philipodonnell »

linkregister wrote: post #161
philipodonnell wrote: post #117

Earlier quoted context omitted.

Terms of service. It is not considered a contract if you are just an anonymous user, but once you take an affirmative action like creating an account you're explicitly agreeing to that and they are legally binding. The parent comment did not specify anonymous and plenty of bots create accounts to scrape with. But even aside from the explicit contractual terms, even without one, you cannot just run around acting in ba…

Since when do anonymous users not have to adhere to publicly-stated terms of service?

Short answer is, they never were. You can't just leave a contract around and say that anyone who gets nearby and/or reads it is now bound by it. It requires some type of affirmative act to acknowledge that you read and understand the contract and are voluntarily entering into a formal relationship. That's what that "I have read and agree to the terms of service" checkbox on most sites is for.

Otherwise I could leave a piece of paper in a coffee shop saying. "Apple stock will go up 5% in the next month - By reading these words you agree not to trade Apple stock in the next month unless you pay me a royalty.". Reading something does not mean agreeing to its terms and if the content does not require agreeing to its terms to access the content, then the terms are may as well not exist until you do.

It doesn't mean you can ignore everything there. If you reproduce the content then you are still liable under copyright laws. If you DDOS the site, you are still liable under tortious interference laws.

But outside of something covered by other laws, there is no legal recourse to just putting "no bots" in your terms of service and then suing any bots you find. You can try to _block_ bots, just like you can hide your sheet of paper when you see a cop getting coffee, but you can't sue them for breaking terms they never agreed to. I don't believe there is actually any case law yet as to whether a bot can even agree to a contract. For evidence you can look at the recent LinkedIn case, which, while it is on appeal, turned on this issue.

Re: It is not possible to detect and block Chrome headless

#170

Post by nybble41 »

JeremyBanks wrote: post #118
nybble41 wrote: post #89

Earlier quoted context omitted.

Regarding YouTube in particular, I tend to open up videos in background tabs for later viewing and find it very annoying that they start playing automatically before I get around to that tab. I did go there to watch the video—eventually. Just not the second that the page finishes loading. YMMV. A persistent setting to enable or disable auto-play would be ideal.

Chrome doesn't autoplay videos in background tabs until you focus them.

Actually it doesn't even seem to load most of the page in a background tab until it's focused... which is also annoying. (Or perhaps it's just the parts hidden behind 'onload' JavaScript, which these days is most of the page content.) Part of the reason for opening the tab in the background is getting the lengthy loading process out of the way while I'm reading something else.

For this reason I tend to activate the tab and then go back to what I was doing before... and then the video auto-plays.

Post reply on HN