Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

311–312 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#311

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Check if your electricity provider offers Green Button Download. https://www.energy.gov/data/green-button

Seconded. My company offers this and it's a great resource.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#312
I am running a no-code web automation and data extraction tool called https://automatio.co. And from my experience most of the time when using quality residential proxies you will be fine. But that comes at cost since they are way expensive then data center proxies.

But for some websites, even residential ips doesn't let you pass.

I noticed there is like a premium reCaptcha service, which just work differently then standard one and not let you pass. It's mostly shown with a Cloud flare anti bot page.

Post reply on HN