Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

221–230 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#221

For the row "Long-lived sessions after sign-in" the author mentions that this solution is for social media automation i.e. you build a tool to automate social media accounts to manage ads more efficiently. I am curious by what the author means by automating social media accounts to manage ads more efficiently

Clickfraud

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#222

Earlier quoted context omitted.

Yeah, there’s a central service that all Flight search is connected to, irregardless of airline. The airlines are charged per search to that api, so they monitor their ”look to book” rationvery closely. That ratio remains quite stable im the absence of bots, but skyrockets with any bot activity. Hence, they know from that metric how big of a bot problem they have and how much money they are losing. Major flight searc…

In fact the airlines are charged per book, but if and only if the look to book stays within reasonable bounds. If it rockets up, they’re on the hook for the penalties

Thanks for the clarification! I probably misremember some of the details.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#223

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

At a minimum any scraper that doesn't execute JS needs to impersonate a screen reader user agent. Locking out disabled people has to be many levels of illegal in most countries.

As a disabled person myself I will go as far to suggest that the websites should allow unfettered bot access to the disabled i.e. anything a user without malicious intent is allowed to do on a platform should be allowed to be done by a bot for a disabled because accessibility & equity are a joke.

Social media platforms has made physical appearance as the first class citizen of the reputation economy. I'm not even talking about those platforms which outright bury content from the disabled as a policy, I'm talking about those platforms whose algorithms favor selfies, videos over text/URLs and thereby putting those with accessibility issues in severe disadvantage.

Why would you use such platforms one might say, Do something which has nothing to do with the reputation economy they might add; Well have you looked at LinkedIn lately? LinkedIn has become ubiquitous with professional job search and 30 second video intro is the very first thing on the profile, not the skills which the platform was meant to be when it was launched. One must be naive to claim that the physical appearance on that video or profile picture doesn't affect the job prospects(Several studies have stated otherwise).

It's not just the physical appearance, The action of creating videos or posting photos itself is hard as a time-constrained person[1] and so I think it's reasonable to ask the platform to allow bots to post deep-fake videos of the user doing silly things which these platform expects from an average user.

[1] https://abishekmuthian.com/time-constrained-person/

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#225
post #155

Earlier quoted context omitted.

thats great till you're in a foreign country and your phone suddnely decides to die leaving you stranded and unable to access bank accounts or prove your identity. (happened to me)

2FA isn't limited to one device, or specific 2FA mobile apps. For example I use oathtool for most 2FA things; you just need to store the secret (often in the form of a QR code, but many services will also offer a text version, and if not you can decode the QR). 100% reliance on a phone which is easily lost, broken, stolen, etc. without backup is really bad IMO. My bank (Revolut) only had a mobile app, and no way to c…

Revolut now has a web app [1], which still tries to get you to log in via the mobile app but this is not necessary. So long as you know your pass code and have alternative access to your email then you can log in and do most of the things you can do via the app. You do have to wait 10 seconds for the privilege though (before allowing access via email there is a timer before you can confirm you do not have access to the mobile app.).

1. https://blog.revolut.com/introducing-the-revolut-web-app/

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#226
post #206

Earlier quoted context omitted.

> -validating stolen credit card details To be clear, "validating" is an industry euphemism for stealing, just for a different purpose. How do you validate the card is live? Run a real transaction through it and mark it based on the result. But what do you run for this real transaction? Well, whatever you want. Typically it'll be something to avoid suspicion as much as possible, but the thief gets to pick what they t…

Not exactly. Some people are in the business of gathering and selling valid credit cards. They won't cash out on them or buy items. Instead, they'll collect cards from a source (skimming, hacking, whatever), validate them by adding them to a website that does an authorization (those $1 checks that never get committed). They can then sell them wholesale for a premium compared to non-verified cards.

You'd think so, but I've personally had to find ways of blocking people who were buying premium services for an online service to validate cards.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#227
post #154
post #146

Earlier quoted context omitted.

Selenium and other tools in that class essentially just build an api on top of a standard consumer browser engine(s). There are some differences that are difficult to completely hide, but it’s about as close to real as it gets and can be very difficult if not impossible to tell it’s an automation framework vs a standard web browser. Travel information is also one of those services where it’s not weird for a significa…

By default Selenium exposes a few things in JS that are pretty trivial to detect, so you need to disable/hide that for starters. I don't know how easy or hard that is, but stock Selenium is a poor way to get around anti-bot stuff.

Its fairly trivial, some stuff needs a modified browser executable or some JavaScript magic but it wont take you longer than 2-5 Hours to bypass most of the heuristics, disabling the window.navigator.webdriver flag and getting a residential IP is on its own enough to get single click captchas every 1-2 tries for example. To be fair I haven't looked into it since 2019 but i doubt that its gotten much harder.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#228
post #54

Earlier quoted context omitted.

This! Mobile IPs are far more lucrative. Many services will drop captchas and other anti-bot stuff for consumer mobile IPs. I recall Plaid at some point would run their bank scraping through mobile IPs. This sketchy company lets mobile app developers monetize user base by letting other people pay $$ to route requests through random people’s mobile IPs: https://brightdata.com/

Brightdata is formerly known as as Luminati who is owned by same company as Hola VPN. Similarly NordVPN owns Oxylabs (who mostly hack routers and cameras and sell those as residential IP’s).

But they have a white paper on how ethical their proxies are! https://oxylabs.io/Oxylabs_Residential_Proxy_Acquisition_Han...

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#229

Earlier quoted context omitted.

You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…

>I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. Have you tried not using these things? Anonymity is exactly what bots want. They want to be able to post a spam message every single second and be impossible to ban since they are anonymous. The internet can't function if people are allowed to be anonymous.

> The internet can't function if people are allowed to be anonymous.

You must have missed the first 20 or so years of its existence, if that's your position.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#230
post #8
post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

2FA should be a requirement on everything now. And if your site can't for some reason or you don't want to deal with it, then limit your site to external login providers only. 2FA, especially app based, has been proven to work really really well.

Hard disagree. A recipe sharing website or food delivery service DOES NOT require as much security as my email or bank account, and never will.
Post reply on HN