Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

251–260 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#251

Post by orf »

selfhoster11 wrote: post #229
charcircuit wrote: post #141

Earlier quoted context omitted.

>I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. Have you tried not using these things? Anonymity is exactly what bots want. They want to be able to post a spam message every single second and be impossible to ban since they are anonymous. The internet can't function if people are allowed to be anonymous.

> The internet can't function if people are allowed to be anonymous. You must have missed the first 20 or so years of its existence, if that's your position.

It was much smaller, and spam messages where everywhere.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#252

Post by oxymoron »

azalemeth wrote: post #106
oxymoron wrote: post #15

Earlier quoted context omitted.

Yeah, I used to work for one of the major anti-bot vendors. Customers weren't clueless. Nobody buys these solutions because they're so much fun, it's a cost center and they monitor their ROI quite closely. Credit card charge backs, impact to infrastructure, extra incurred cost due to underlying api's (like in the Airline industry in particular) etc are all reasons why bot mitigation is a better option than nothing fo…

You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…

Different customers have different attitudes towards this. Some of them are _very_ focused on conversion and will disable anything which causes additional user friction. For others, the economic damage of bots is just so painful that it makes economic sense for them to add friction for a few percent of users.

I'm a linux user myself, so I know for a fact that neither my previous employer, nor other bot vendors, will block linux user agents in particular. Customers generally don't mind a universal requirement for JS execution, so that's just a fact of life. We generally did try to avoid blocking privacy focused browsers, though. We certainly monitored false positive rates and knew pretty well how we affected users.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#253

Post by arp242 »

iechoz6H wrote: post #225
arp242 wrote: post #155

Earlier quoted context omitted.

2FA isn't limited to one device, or specific 2FA mobile apps. For example I use oathtool for most 2FA things; you just need to store the secret (often in the form of a QR code, but many services will also offer a text version, and if not you can decode the QR). 100% reliance on a phone which is easily lost, broken, stolen, etc. without backup is really bad IMO. My bank (Revolut) only had a mobile app, and no way to c…

Revolut now has a web app [1], which still tries to get you to log in via the mobile app but this is not necessary. So long as you know your pass code and have alternative access to your email then you can log in and do most of the things you can do via the app. You do have to wait 10 seconds for the privilege though (before allowing access via email there is a timer before you can confirm you do not have access to t…

The web app doesn't allow basic things like transferring money. It's borderline useless.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#254

Post by krageon »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

> he'd probably take down the whole repository.

I know how bad this issue is, and I wouldn't take down this repository. Anti-bot software does not work, anyone who pays 10m per year to have it simply has too much money.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#255

Post by todd8 »

ensiferum wrote: post #241
fragmede wrote: post #157

Earlier quoted context omitted.

> -validating stolen credit card details To be clear, "validating" is an industry euphemism for stealing, just for a different purpose. How do you validate the card is live? Run a real transaction through it and mark it based on the result. But what do you run for this real transaction? Well, whatever you want. Typically it'll be something to avoid suspicion as much as possible, but the thief gets to pick what they t…

My bank called as they had red flagged a suspicious charge on on my credit card. The sum was $0. The bank rep told me that this often flies under the radar and doesn't show up in some records and it indeed did not show up in my transaction history that i could see online in my banking details. But yeah the point was exactly the same. The fraudsters testing whether the charge goes through and the card is alive/valid.

Did you verify that it really was the bank calling and not a scammer? I get calls and texts from “credit card fraud departments”, “banks”, “service warranty departments”, “Amazon billing”, “Microsoft security”, “Social Security Administration”, “IRS”, and others frequently; 95% are scammers.

The most amusing to me are the ones from “Microsoft” to alert me that they have detected malware on my computer.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#256

Post by cebert »

Mister_Snuggles wrote: post #151
cebert wrote: post #133

Earlier quoted context omitted.

Have you considered using Playwright to automate that instead?

I had never heard of this, but it looks like a reasonable option. My go-to for this type of thing would be Python+Selenium+Firefox, but only due to familiarity with those.

Playwright is easy to get started with. The even tools that allow you to record your browser actions and covert it into code ( https://playwright.dev/ ).

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#257

Post by Tor3 »

walrus01 wrote: post #59
Mister_Snuggles wrote: post #31

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.

This also depends on the country. Where I live (Europe) the rate now changes by the minute, or thereabouts. That was made possible after everybody had to change to wireless meters. Sometimes you'll get a warning in advance - a newspaper may write "If you live here or here, don't do your cooking at this particular hour". Some providers still have fixed rate options, some apparently don't. What I dislike the most is that they're trying to force us to run our washing during the night, something the insurance company and the fire department warn intensely against. And I don't want to be at sleep if a fire starts (which happens here and there, through the year). But that's what the pricing scheme tries to enforce.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#258

Post by Tor3 »

fragmede wrote: post #180
azalemeth wrote: post #106

Earlier quoted context omitted.

You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…

You could always go into your local bank branch instead of accessing it over the Internet. Your desktop's accelerometer helps add to your computers 'run by a human' score. Normally I'd take more issue with whatever possible privacy issue there, but my bank is where I keep my money so I'm really okay with them trying hard to keep bots out of my account.

The physical presence of banks is going away. Where I live you can't do any kind of monetary transaction in the local branch offices of any of the banks anymore. You can a) apply for a loan (and even that may go away soon), and b) identify yourself and get a physical token used for accessing the bank via the net. You can't withdraw money, you can't pay bills, you can't exchange currency. I haven't been inside my bank for many years, there's just nothing I can do there. The last time I visited the bank was with my wife (an immigrant), with her documents, to get her into the system.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#259

Post by timthorn »

walrus01 wrote: post #59
Mister_Snuggles wrote: post #31

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.

Octopus Energy in the UK has a tariff that charges half-hourly rates. They also offer an API to interrogate current pricing and usage and encourage their customers - including domestic users - to take advantage of it: https://developer.octopus.energy/docs/api/

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#260

Post by sokoloff »

Tor3 wrote: post #257
walrus01 wrote: post #59

Earlier quoted context omitted.

Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.

This also depends on the country. Where I live (Europe) the rate now changes by the minute, or thereabouts. That was made possible after everybody had to change to wireless meters. Sometimes you'll get a warning in advance - a newspaper may write "If you live here or here, don't do your cooking at this particular hour". Some providers still have fixed rate options, some apparently don't. What I dislike the most is th…

I’m curious to see the stats they are relying on and the communications materials the fire department and insurance company are using on this topic.

It seems to me from a life-safety angle that their energy would likely be far better spent on recommending smoke alarms, CO meters, and periodic cleaning of dryer vents than on recommendations against sleeping with washing/drying machines running.

Post reply on HN