Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

31–40 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#31

Post by Mister_Snuggles »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

I wish they'd limit it to just stopping credential stuffing.

Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving up some JavaScript to check my browser.

The information I'm trying to get is public and can be accessed without any kind of authentication. I'm willing to bet that they flipped the anti-bot stuff on their load balancer on for the entire site instead of doing the extra work to only enable it for just electricitycompany.com/myaccount/ (where you do have to log in).

I also asked the company if they'd be willing/able to push the power rates out via the smart meters so that my interface box (Eagle-200) could pick it up, they said they have no plans to do so.

The next step is to scrape the web site for the provincial power regulator, which shows the power rates for each provider. Of course, the regulator's site has different issues (rounding, in particular), I haven't dug any further to see if I can make use of this.

All of this effort to get public information in an automated fashion.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#32

Post by [deleted] »

curun1r wrote: post #17

There’s one technique that can be very useful in some circumstances that isn’t mentioned. Put simply, some sites try to block all bots except for those from the major search engines. They don’t want their content scraped, but they want the traffic that comes from search. In those cases, it’s often possible to scrape the search engines instead using specialized queries designed to get the content you want into the blu…

[deleted]

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#33

Post by bredren »

melony wrote: post #3
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

The gold standard is residential IP. It is not cheap but its effectiveness is indisputable.

Unless you use a residential ip proxy network.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#34

Post by judge2020 »

Mister_Snuggles wrote: post #31
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

> I wish they'd limit it to just stopping credential stuffing.

A product protecting against credential stuffing might as well prevent denial-of-service as well.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#35

Post by ev1 »

yamakadi wrote: post #12
IceWreck wrote: post #10

Half of the short-links to cutt.ly aren't working. Why use short links in markdown ?

It’s most likely for tracking clicks. Better to just search for the company names instead of clicking on the links in case they lead to unexpected places.

It's hidden affiliate spam without disclosure.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#36

Post by marginalia_nu »

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general.

Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching some resource that is expensive that bots in general refuse to lay off. Bots with seconds between the requests rarely get blocked even by CDNs.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#37

Post by kevin_thibedeau »

Mister_Snuggles wrote: post #31
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

At a minimum any scraper that doesn't execute JS needs to impersonate a screen reader user agent. Locking out disabled people has to be many levels of illegal in most countries.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#38

Post by marginalia_nu »

curun1r wrote: post #25
amelius wrote: post #18

Earlier quoted context omitted.

But won't the search engines block you after some limit has been reached?

Eventually, but they’re not very aggressive when it comes to bot detection. Simple IP rotation usually works.

I ended up having to put my search engine behind a CDN and CAPTCHA every new IP because I got something like 30,000 search requests like this _per_hour_ from a botnet. They seem to have backed off now but they were really quite persistent for a while.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#39

Post by nocturnial »

I knew there was a reason why I used client certificates and alternate ports.

Why is it so difficult to just respect robots.txt? Maybe there's an idea for a browser plugin that determines if you can easily scrape the data or not. If not, then the website is blocked and then traffic will drop. I know this is a naive idea...

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#40

Post by soheil »

marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

Google can access any site without being blocked. They dominate the search space and give little incentive for site owners to allow other bots. I'd say bypassing these measures is fair game while there is a monopoly in search space. We don't want a web that only Google can access.

By the way great work on Marginalia search engine, I love it.

Post reply on HN