Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

141–150 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#141

Post by charcircuit »

azalemeth wrote: post #106
oxymoron wrote: post #15

Earlier quoted context omitted.

Yeah, I used to work for one of the major anti-bot vendors. Customers weren't clueless. Nobody buys these solutions because they're so much fun, it's a cost center and they monitor their ROI quite closely. Credit card charge backs, impact to infrastructure, extra incurred cost due to underlying api's (like in the Airline industry in particular) etc are all reasons why bot mitigation is a better option than nothing fo…

You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…

>I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech.

Have you tried not using these things? Anonymity is exactly what bots want. They want to be able to post a spam message every single second and be impossible to ban since they are anonymous. The internet can't function if people are allowed to be anonymous.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#142

Post by remram »

nocturnial wrote: post #39

I knew there was a reason why I used client certificates and alternate ports. Why is it so difficult to just respect robots.txt? Maybe there's an idea for a browser plugin that determines if you can easily scrape the data or not. If not, then the website is blocked and then traffic will drop. I know this is a naive idea...

I don't understand what you recommend. Who would drop the traffic?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#143

Post by Gigachad »

ixs wrote: post #30
Gigachad wrote: post #8

Earlier quoted context omitted.

2FA should be a requirement on everything now. And if your site can't for some reason or you don't want to deal with it, then limit your site to external login providers only. 2FA, especially app based, has been proven to work really really well.

It does not. There are myriad ways of extracting the TOTP seed from these apps... Or you just reverse engineer the setup/confirmation process and then you can generate/trigger your own tokens from your automation workflow. 2FA is a good security feature but it does not help against web scraping. Credential stuffing and other 3rd party attacks? Yes, it _can_ help. But it does not always help. There's a phishing group…

Having malware installed on every users phone is so many orders of magnitude harder than downloading the latest db dump and testing the email/password on every other site.

At the bare minimum, TFA stops most attacks. That's a whole lot better than the current situation.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#144

Post by xboxnolifes »

calvinmorrison wrote: post #122
toomuchtodo wrote: post #75

Earlier quoted context omitted.

A link is generated, emailed to the user, and clicking the link logs them in.

So SSO but you have to trust the email provider instead of another random SaaS

Basically. Most websites already make you login with an email and verify you have access to that email and use the email as a password recovery mechanism. May as well just use the email itself as the login.

Of course if everyone did this, then all of your logins would have the same password (your email login).

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#145

Post by gjs278 »

marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

it’s not illegal to scrape a website

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#146

Post by xnyan »

Jugurtha wrote: post #93
Aperocky wrote: post #81

Earlier quoted context omitted.

Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.

> Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila. Can you say more about this? What do you mean by "Once you get to selenium it's usually over", and how do you manage cold starts in Selenium and emulating heavy usage? Say your program starts right now, I assume you don't go through "adding heavy usage" to "warm-up", then get down to business, corre…

Selenium and other tools in that class essentially just build an api on top of a standard consumer browser engine(s). There are some differences that are difficult to completely hide, but it’s about as close to real as it gets and can be very difficult if not impossible to tell it’s an automation framework vs a standard web browser.

Travel information is also one of those services where it’s not weird for a significant number of their users to use it quite heavily, making behavioral detection more difficult.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#147

Post by anthuswilliams »

jrochkind1 wrote: post #131
nieve wrote: post #121

Earlier quoted context omitted.

I.e. what Facebook does if you don't log in for long enough. Two days ago I got a pair of messages to the same address with links to completely bypass login and verbiage about how they'd seen I was having trouble logging in followed an sms message with the same to a phone number they're not supposed to be using. It looks a lot like phishing, but it comes out of Facebook's servers and they've done it to me before.

> to a phone number they're not supposed to be using What do you mean?

I'm not the poster you're replying to, but: Facebook collects asks for your phone number for security/account recovery reasons, but then turns around and uses it to market to you.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#148

Post by temp8964 »

Dah00n wrote: post #51
TruthWillHurt wrote: post #42

Earlier quoted context omitted.

You're forgetting a case where a website offers a garbage API that doesn't provide all the data available via the web interface, either due to neglegance (Apple Store developer console), Security (Google Play Store accounting data), Or financial gain (AppsFlyer "premium API").

That doesn't change anything. The site owner decides what you can and cannot do. If a badly made API meant you could do anything you wanted to then everything could be done to those sites running them. That is not how this work. Outside normal usage you need permission.

I really don't believe "The site owner decides what you can and cannot do." statement. What is the base of this? This does not seem to apply to anything in the real world.

In most cases you have very limited ability to decide what other people cannot do. And other people has mostly infinite choices of what they can do. I never heard anything as broad as you said. What you said is like a person standing on the street with a T-shirt says "do not look at me more than twice" and claim it has a legal binding to the whole world.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#149

Post by Mister_Snuggles »

walrus01 wrote: post #59
Mister_Snuggles wrote: post #31

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.

In Alberta, the electricity system has been deregulated so you can buy from numerous providers. The Utilities Consumer Advocate shows 187[0] different electricity plans available in my city. My currently plan and provider changes rates monthly, but some providers allow you to sign up for 3-year or 5-year fixed-rate plans.

Thanks to deregulation, the electricity rate isn't the only thing you pay for though. There is also a Transmission Charge, Distribution Charge, and Local Access Fee. These are all per-kWh charges and change very rarely.

In October, my electricity rate is $0.10730/kWh, but my total cost is actually $0.16346/kWh plus the per-day charge ($0.202/day). Tomorrow the November rate will be published.

[0] https://ucahelps.alberta.ca/cost-comparison-tool-result.aspx...

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#150

Post by Mister_Snuggles »

xnyan wrote: post #136
Mister_Snuggles wrote: post #31

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Unfortunately, the days of reliable non-JavaScript capable scraping are over. Fortunately there are plenty of tools to handle this, and at a hobby level not particularly resource intensive. Something like this is simple and reliable in many cases: https://github.com/berstend/puppeteer-extra/tree/master/pack...

I will take a look at this - thank you!
Post reply on HN