Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

131–140 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#131

Post by jrochkind1 »

nieve wrote: post #121
toomuchtodo wrote: post #75

Earlier quoted context omitted.

A link is generated, emailed to the user, and clicking the link logs them in.

I.e. what Facebook does if you don't log in for long enough. Two days ago I got a pair of messages to the same address with links to completely bypass login and verbiage about how they'd seen I was having trouble logging in followed an sms message with the same to a phone number they're not supposed to be using. It looks a lot like phishing, but it comes out of Facebook's servers and they've done it to me before.

> to a phone number they're not supposed to be using

What do you mean?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#132

Post by judge2020 »

kevin_thibedeau wrote: post #37
Mister_Snuggles wrote: post #31

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

At a minimum any scraper that doesn't execute JS needs to impersonate a screen reader user agent. Locking out disabled people has to be many levels of illegal in most countries.

Blocking for not supporting JS isn't illegal nor a violation of the US ADA. You can add requirements for disabled people to access your services as long as it's reasonable, and the prevalence of screen readers that work with JS turned on likely qualifies requiring JS a reasonable request. It'd be like saying "you can't deny someone using a IE8 screen reader by only offering TLS 1.3".

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#133

Post by cebert »

Mister_Snuggles wrote: post #31
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Have you considered using Playwright to automate that instead?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#134

Post by spijdar »

azalemeth wrote: post #106
oxymoron wrote: post #15

Earlier quoted context omitted.

Yeah, I used to work for one of the major anti-bot vendors. Customers weren't clueless. Nobody buys these solutions because they're so much fun, it's a cost center and they monitor their ROI quite closely. Credit card charge backs, impact to infrastructure, extra incurred cost due to underlying api's (like in the Airline industry in particular) etc are all reasons why bot mitigation is a better option than nothing fo…

You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…

> I really hope that the clients you mention know that they piss off a proportion of their users with every move they take.

With all due respect, if the tech can make a large impact on the problems mentioned above, I'm sure it's an easy decision for the big companies to take decimating bot activity over the tiny minority of users who proactively decide to disable JavaScript.

Said as someone who uses NoScript, FWIW.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#135

Post by chucksmash »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

The credential stuffing wiki page didn't exist the last time I thought about invalid traffic so I'm pretty out of date.

How is there not an equilibrium here that cuts off credential stuffers? I'd naively imagine the residential IP providers have some measure of bad actors they themselves use to determine if a client is worth it, and that someone getting all your IPs blacklisted would get dropped pretty quickly.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#136

Post by xnyan »

Mister_Snuggles wrote: post #31
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Unfortunately, the days of reliable non-JavaScript capable scraping are over.

Fortunately there are plenty of tools to handle this, and at a hobby level not particularly resource intensive. Something like this is simple and reliable in many cases: https://github.com/berstend/puppeteer-extra/tree/master/pack...

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#137

Post by gzer0 »

abadger9 wrote: post #102

I'm a lead engineer on the search team of a publicly traded company who's bread and butter is this domain. I was curious about this list, it candidly misses the mark- the tech mentioned in this blog is what you might get if you hired a competent consultant to build out a service without having domain knowledge. In my experience, what's being used on the bleeding edge is two steps ahead of this.

I have a considerable amount of experience in the industry.

Some of these so-called "advanced" techniques:

  * We use our own mobile emulation software (similiar to bluestacks). Turns out, mobile helps with a lot of things (below).
  * We use mobile IPs only. Mobile LTE data users are behind CGNATfor IPV4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point.
  * All you need is a new useragent and browser fingerprint; combined with emulation + mobile IPs, there's really no easy way for companies to block this.
  * With the advent and ease of virtualization; we avoid using any headless browsers. Seriously, if you can, never use headless. This should be close to rule number one for anyone looking to operate any kind of scrapers. All of our scrapers are run in isolated virtual instances with full mobile browsers.
  * We can easily reset our device identifier, device carrier, simulated SIM information, and especially important is the Google advertising ID that is set per device; the list goes on. The key here is #1, our mobile emulation software.
  * Our automation scripts are a combination of human recorded set of actions which we then perfected and can run in certain loops (for some of our data).

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#138

Post by judge2020 »

chucksmash wrote: post #135
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

The credential stuffing wiki page didn't exist the last time I thought about invalid traffic so I'm pretty out of date. How is there not an equilibrium here that cuts off credential stuffers? I'd naively imagine the residential IP providers have some measure of bad actors they themselves use to determine if a client is worth it, and that someone getting all your IPs blacklisted would get dropped pretty quickly.

In reality residential US ISPs don't really care if their users are getting a sub-par experience since they're often the only fast/fiber provider in the area of their customers, meaning customers have no way to switch. Plus, when a website doesn't work, unless the page itself calls out the ISP (which they never do), customers will think it's an issue with the website and won't possibly attribute blame to their ISP until they're deep in forum threads with people telling them "it's probably your ISP not doing anything about bad customers" - the amount of users going so far to learn this information, then accepting it, is extremely low.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#139

Post by Lukabuz »

xiamx wrote: post #26
rp1 wrote: post #20

It's very easy to install Chrome on a linux box and launch it with a whitelisted extension. You can run Xorg using the dummy driver and get a full Chrome instance (i.e. not headless). You can even enable the DevTools API programmatically. I don't see how this would be detectable, and probably a lot safer than downloading a random browser package from an unknown developer.

Try your technique on a few of these fingerprint testing sites https://github.com/niespodd/browser-fingerprinting#fingerpri... I'm pretty sure it's quite detectible

If I am correct in assuming the parent is talking about puppeteer, there is a plugin[1] that claims to evade most of the methods used to detect headless browsers. I have used it recently for just that purpose, and I can say that it worked wifh minimal setup and configuration for my usecase, but I guess depending on the detection mechanisms youre evading YMMV.

The creator of that plugin does mention it is very much a cat and mouse game, just like most of the “scraping industry”

https://www.npmjs.com/package/puppeteer-extra-plugin-stealth

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#140

Post by hbgl »

marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

Technically it should be illegal to scrape websites without the consent of the server owner because it would be a violation of their property rights. That being said, I think that in reality it would be pretty difficult to get courts to agree with you and then to enforce it.
Post reply on HN