Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

81–90 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#81

Post by Aperocky »

northwest65 wrote: post #5
melony wrote: post #3

Earlier quoted context omitted.

The gold standard is residential IP. It is not cheap but its effectiveness is indisputable.

Back when we had to scrape airline websites to get the deals they withheld for themselves, residential IP was indeed the way. Once the cottoned on to it and blocked id, you'd simply cycle the ADSL model, get a new IP, and off you'd go again. Now the best part... one division (big team) of our company worked for the (national carrier) airline , one division of our company worked for the resellers (we had a single grad…

Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#83

Post by marginalia_nu »

Aeolun wrote: post #77
marginalia_nu wrote: post #65

Earlier quoted context omitted.

There are laws against unauthorized computer access. This is a scenario where you have a server explicitly saying "Stop! You are not permitted to access this computer!", and yet you persist in circumventing that by hiding your identity and accessing it anyway. Those are some murky waters.

The server says nothing of the kind. The response that was previously positive is now broken, and it happens to be fixed if you access it from a different IP. Maybe we need a status code that means ‘lay off all the requests made from this entire system’?

How do you interpret a

401 Unauthorized

to mean you are authorized to access the resource?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#84

Post by connectsnk »

For the row "Long-lived sessions after sign-in" the author mentions that this solution is for social media automation i.e. you build a tool to automate social media accounts to manage ads more efficiently.

I am curious by what the author means by automating social media accounts to manage ads more efficiently

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#85

Post by mindslight »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

If your password db is so broken that it's useful to create a term to abstract attacks ("credential stuffing"), then the right answer is to actually fix that security (eg pick users passwords for them, or completely replace with email auth), rather than thinking you're raising the bar by requiring attackers to come from a residential IP.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#86

Post by boolemancer »

Dah00n wrote: post #51
TruthWillHurt wrote: post #42

Earlier quoted context omitted.

You're forgetting a case where a website offers a garbage API that doesn't provide all the data available via the web interface, either due to neglegance (Apple Store developer console), Security (Google Play Store accounting data), Or financial gain (AppsFlyer "premium API").

That doesn't change anything. The site owner decides what you can and cannot do. If a badly made API meant you could do anything you wanted to then everything could be done to those sites running them. That is not how this work. Outside normal usage you need permission.

If I can access the information in Chrome, I should be able to access the information with my own client, and the site operator shouldn't have any say in that.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#87

Post by nocturnial »

kodah wrote: post #78
ChuckMcM wrote: post #43

I am always amazed when otherwise intelligent people assert without data that the marginal cost of serving web traffic to scrapers/bots is zero. It is kind of like people who say "Why don't they put more fuel in the rocket so it can get all the way into orbit with just one stage?" It sounds great but it is a completely ignorant thing to say.

When I worked in e-commerce as a SRE, bots were doing two things: - trying to disrupt business processes (eg: false referral listings, gift card scams, etc) - trying to disrupt systems I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs tha…

> I'm sure there are folks who use bots and scrapers for home automation

I know this is off-topic, but I'm really curious. How does scraping the web help with home automation? Maybe downloading weather data could help, but crawling the web? I think I'm missing something about home automation.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#88

Post by dpryden »

It always amazes me how people believe they have a right to retrive data from a website. The HTTP protocol calls it a request for a reason: you are asking for data. The server is allowed to say no, for any reason it likes, even a reason you don't agree with.

This whole field of scraping and anti-bot technology is an arms race: one side gets better at something, the other side gets better at countering it. An arms race benefits no one but the arms dealers.

If we translate this behavior into the real world, it ends up looking like https://xkcd.com/1499

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#89

Post by irrational »

walrus01 wrote: post #59
Mister_Snuggles wrote: post #31

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.

I’ve never seen such a thing. Every locale here has a meter and you are charged monthly for exactly how much you use. Just like water.
Post reply on HN