Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

151–160 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#151

Post by Mister_Snuggles »

cebert wrote: post #133
Mister_Snuggles wrote: post #31

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Have you considered using Playwright to automate that instead?

I had never heard of this, but it looks like a reasonable option. My go-to for this type of thing would be Python+Selenium+Firefox, but only due to familiarity with those.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#152

Post by [deleted] »

Mister_Snuggles wrote: post #31
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

[deleted]

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#153

Post by welanes »

Another great resource is incolumitas.com. A list of detection methods are here: https://bot.incolumitas.com/

I run a no-code web scraper (https://simplescraper.io) and we test against these.

Having scraped million of webpages, I find dynamic CSS selectors a bigger time sink than most anti-scraping tech encountered so far (if your goal is to extract structured data).

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#154

Post by arp242 »

xnyan wrote: post #146
Jugurtha wrote: post #93

Earlier quoted context omitted.

> Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila. Can you say more about this? What do you mean by "Once you get to selenium it's usually over", and how do you manage cold starts in Selenium and emulating heavy usage? Say your program starts right now, I assume you don't go through "adding heavy usage" to "warm-up", then get down to business, corre…

Selenium and other tools in that class essentially just build an api on top of a standard consumer browser engine(s). There are some differences that are difficult to completely hide, but it’s about as close to real as it gets and can be very difficult if not impossible to tell it’s an automation framework vs a standard web browser. Travel information is also one of those services where it’s not weird for a significa…

By default Selenium exposes a few things in JS that are pretty trivial to detect, so you need to disable/hide that for starters. I don't know how easy or hard that is, but stock Selenium is a poor way to get around anti-bot stuff.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#155

Post by arp242 »

cultofmetatron wrote: post #79
Gigachad wrote: post #8

Earlier quoted context omitted.

2FA should be a requirement on everything now. And if your site can't for some reason or you don't want to deal with it, then limit your site to external login providers only. 2FA, especially app based, has been proven to work really really well.

thats great till you're in a foreign country and your phone suddnely decides to die leaving you stranded and unable to access bank accounts or prove your identity. (happened to me)

2FA isn't limited to one device, or specific 2FA mobile apps. For example I use oathtool for most 2FA things; you just need to store the secret (often in the form of a QR code, but many services will also offer a text version, and if not you can decode the QR).

100% reliance on a phone which is easily lost, broken, stolen, etc. without backup is really bad IMO. My bank (Revolut) only had a mobile app, and no way to contact them outside of it (I tried...) I need to switch banks.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#156

Post by gwbas1c »

abadger9 wrote: post #115
ryandrake wrote: post #110

Earlier quoted context omitted.

Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.

Sorry about that, we're unable to discuss our projects with adjacent teams within the company. What you're saying is a valid frustration, the motivation behind the original comment was to put a thermometer on the repo.

Well, don't go getting our hopes up!

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#157

Post by fragmede »

jameshart wrote: post #116
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

Bots aren't just trying credential stuffing. They are: - committing clickfraud to game ad and referral revenue systems - posting fake or spam reviews and comments - generating fake behavioral signals to help bypass CAPTCHAs to help create accounts on other sites that can post spam comments - validating stolen credit card details - screwing with your metrics collection if you can't identify them as bots All of that is…

> -validating stolen credit card details

To be clear, "validating" is an industry euphemism for stealing, just for a different purpose. How do you validate the card is live? Run a real transaction through it and mark it based on the result. But what do you run for this real transaction? Well, whatever you want. Typically it'll be something to avoid suspicion as much as possible, but the thief gets to pick what they test it with, so why not pick something that they'll personally benefit from? There are so many online games with purchasable currency these days, it's hard to choose.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#158

Post by arp242 »

temp8964 wrote: post #148
Dah00n wrote: post #51

Earlier quoted context omitted.

That doesn't change anything. The site owner decides what you can and cannot do. If a badly made API meant you could do anything you wanted to then everything could be done to those sites running them. That is not how this work. Outside normal usage you need permission.

I really don't believe "The site owner decides what you can and cannot do." statement. What is the base of this? This does not seem to apply to anything in the real world. In most cases you have very limited ability to decide what other people cannot do. And other people has mostly infinite choices of what they can do. I never heard anything as broad as you said. What you said is like a person standing on the street…

> I really don't believe "The site owner decides what you can and cannot do." statement. What is the base of this? This does not seem to apply to anything in the real world.

It kind of does though; if I own a store and say that only people with hats can enter, then I'm free to do so. Silly? Yes. Legal? Also yes.

There are some circumstances where it's not legal, mostly centred around discrimination. Details on this differ per jurisdiction, but generally speaking you have a right to refuse customers.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#159

Post by arp242 »

peterburkimsher wrote: post #46

2 of my social media accounts have fallen victim to bot detection, despite not using scripts . There are other websites for which I have used scripts, and sometimes ran into CAPTCHA restrictions, but was able to adjust the rate to stay within limits. CouchSurfing blocked me after I manually searched for the number of active hosts in each country (191 searches), and posted the results on Facebook. Basically I question…

CouchSurfing is just shit, full stop. I love the concept and hosted many people, but the way the company has been run over the last few years is beyond atrocious. It's like AirBnB sent over some people to intentionally run it in to the ground or something.

LinkedIn has to deal with a lot of scummy recruiters and scammers; I don't blame them for being very strict.

Post reply on HN