Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It is not possible to detect and block Chrome headless

intoli.com

Post reply on HN
Print view View original

141–150 of 174 posts

Re: It is not possible to detect and block Chrome headless

#141

Post by troels »

dsacco wrote: post #75
troels wrote: post #27

Earlier quoted context omitted.

In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…

> When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Having written bespoke scraping systems professionally, I think you overestimate the applicability of this technique. For one thing, detecting that a scraper is a scraper is the problem, not the prelude to the problem. You might as well block them at that point, if you feel you can reliably dete…

> Having written bespoke scraping systems professionally, I think you overestimate the applicability of this technique.

You may be right, but I would say that it shifts the balance. If the content is poisoned, the scraper will have to spend effort on quality control and they can never know for certain if you have detected their workarounds. It doesn't stop them of course, but it certainly raises the cost of scraping you at little extra cost to you.

Re: It is not possible to detect and block Chrome headless

#142

Post by yorwba »

dsacco wrote: post #140
exikyut wrote: post #134

Earlier quoted context omitted.

> If you want to productively block scrapers, you should consider using a captcha-based system at the application layer, preferably a captcha that hasn't been broken yet and which can't be outsourced to a mechanical turk-based API. If nothing else, doing that will introduce at least 10 - 20 seconds of latency per request, which might be intolerable for many scrapers even if they're quite sophisticated. I'm thinking o…

> Firstly, I'm aware of human-based captcha-defeating systems. You describe captchas that "cannot be outsourced to a mechanical turk-based API". I'm wasn't aware such systems existed, that sounds scary. Do you have any examples? Google's latest captcha specification (and similarly sophisticated systems) must be completed in a small window of time, change rapidly, have a varying number of "rounds", and are extremely a…

> Practically speaking, you can't consistently outsource that to a third party API that uses humans to click and verify the images.

It sounds like you can outsource it, but you essentially need a remote desktop connection for the captcha solver to interact with the page. The overhead might be a bit much, but it doesn't seem impossible.

Re: It is not possible to detect and block Chrome headless

#143

Post by yorwba »

RickHull wrote: post #82
AugustJok wrote: post #52

Earlier quoted context omitted.

I don't see anything wrong with it, I am just impressed the approach.

Evil is an interesting choice of words to mean not wrong and impressive ...

Considering the frequency with which not wrong and impressive things are labeled evil, it is only a matter of time for the word to lose it's negative connotations. Similarly to sick or awesome.

Re: It is not possible to detect and block Chrome headless

#144

Post by est »

odammit wrote: post #121

Blocking crawlers is dead simple: Find a way to build an API for your data that allows you both to make money. Any effort besides that is wasted. Honey pots links? Great my crawler only clicks things that are visible. See capybara. IP thresholds? Great I have burner IPs that hit a good page of yours until I’m blocked (am I time banned, captchad or perma) and then I back that number out across my network of residentia…

I dont stop crawlers, I only randomly feed damaged/wrong data to crawlers.

I especially loving doing this for e-commerce sites. Now the table has turned. Try guess which fraction of your scrapped data were wrong.

Re: It is not possible to detect and block Chrome headless

#145

Post by ThrustVectoring »

mr_toad wrote: post #107
ThrustVectoring wrote: post #30

Earlier quoted context omitted.

There's a similar "analog hole" for video DRM, too.

I wonder how long it will be before someone comes up with the idea of using iPhone style facial recognition to tell whether a human is looking at the TV/Monitor or not.

Oh please no, exposing those sorts of APIs will quickly be utilized by ad-tech guys to make interstitial video ads that don't go away until you finish watching them.

Re: It is not possible to detect and block Chrome headless

#146

Post by odammit »

est wrote: post #144
odammit wrote: post #121

Blocking crawlers is dead simple: Find a way to build an API for your data that allows you both to make money. Any effort besides that is wasted. Honey pots links? Great my crawler only clicks things that are visible. See capybara. IP thresholds? Great I have burner IPs that hit a good page of yours until I’m blocked (am I time banned, captchad or perma) and then I back that number out across my network of residentia…

I dont stop crawlers, I only randomly feed damaged/wrong data to crawlers. I especially loving doing this for e-commerce sites. Now the table has turned. Try guess which fraction of your scrapped data were wrong.

My point is you’re not detecting the ones that you should be the most concerned with.

Re: It is not possible to detect and block Chrome headless

#147

Post by odammit »

littlestymaar wrote: post #131
odammit wrote: post #124

Earlier quoted context omitted.

Caveat to my wasted effort comment: Your'e an e-commerce site that has a problem with people buying goods (especially virtual goods, ebooks, gift cards, etc)[1] with stolen credit cards. You need a solution. The hardest thing I've ever had to crawl (as I mention in another comment in this thread) has been linkedin and Facebook. Why? Because I have to be logged in to get the data I want. If you want to stop crawlers y…

> [1] Why virtual goods? You can't stop shipping or track down the person once the card is reported stolen. In the meantime, virtual good are also zero-cost : when you sell a ebook and the transaction is cancelled by the bank, you didn't lose anything, it's not like the buyer was willing to pay anyway.

Until you see someone do it with buying gift cards. Those people are bastards. In that case using a service that helps detect fraudulent card usage is bueno.

Re: It is not possible to detect and block Chrome headless

#148

Post by toomuchtodo »

averagewall wrote: post #68
nicklaf wrote: post #58

Earlier quoted context omitted.

So record actual user input data and generate similar input patterns stochastically. That said if you try to scale this up beyond what a reasonable, normal user world do in one sitting, you are bound to stand out. Although that said, I find that I trigger such rate-limiting mechanisms already as a human just when searching Google as a human being and clicking through every last search result page.

You'd have to scrape slowly to mimic a real slow user. Maybe at that point you'd be cheaper to get Mechanical Turk to do it. That should solve IP rate limiting, captchas, and just about everything except the endless arms race. Why are so many people going directly to these same-formatted internal URLs without clicking through from random other places? So the site can change the internal URLs and break it all over aga…

You'd use a browser extension, scoped to requests of sites you're interested in, and stream your data back to your infrastructure for processing. You're limited only by your install base and your ingest infrastructure.

Recap [1] does this to extract PACER court documents that are public domain, but access is restricted due to draconian public policy.

[1] https://free.law/recap/

Re: It is not possible to detect and block Chrome headless

#149

Post by civilian »

dsacco wrote: post #76
adrr wrote: post #55

Earlier quoted context omitted.

Poisoning the well is very effective. We employed it at a large ecommerce company that was getting hit by carders testing credit cards on low price point items(sub $5). We were playing cat and mouse with them for six months. Found certain attributes about the browser that the botnet was using and fed them randomized success/fail responses. After two weeks of feeding them bad data, they left and never came back. They…

I think this is a good example of "poisoning the well" in practice, and I was in a similar position as you describe when I was working in incident response at a consumer bank a few years ago. That said, this is a very particular scenario, and I don't think you can generalize the effectiveness of the technique from this example. In situations where attackers are looking for boolean responses, i.e. to verify email addr…

Man. I really like that idea, but is giving a random "success" to a supposed credit card charge PCI compliant?

Re: It is not possible to detect and block Chrome headless

#150

Post by stryk »

adrr wrote: post #55
troels wrote: post #27

Earlier quoted context omitted.

In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…

Poisoning the well is very effective. We employed it at a large ecommerce company that was getting hit by carders testing credit cards on low price point items(sub $5). We were playing cat and mouse with them for six months. Found certain attributes about the browser that the botnet was using and fed them randomized success/fail responses. After two weeks of feeding them bad data, they left and never came back. They…

The whole cat and mouse game thing... for some strange reason that sounds fun to me. Probably because I don't know the details and the workload involved in actually doing it (and it's not my money or inventory at stake). It seems like it would be exciting, in that somewhat naive juvenile-ish fantasy sort of way, to try and figure out how to mitigate the threat, implement it quickly, and deploy it to watch it play out in real time on live production servers. I don't know, maybe I have the wrong idea about the whole thing?
Post reply on HN