Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

51–60 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#51

Post by Dah00n »

TruthWillHurt wrote: post #42
marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

You're forgetting a case where a website offers a garbage API that doesn't provide all the data available via the web interface, either due to neglegance (Apple Store developer console), Security (Google Play Store accounting data), Or financial gain (AppsFlyer "premium API").

That doesn't change anything. The site owner decides what you can and cannot do. If a badly made API meant you could do anything you wanted to then everything could be done to those sites running them. That is not how this work. Outside normal usage you need permission.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#52

Post by ohyeshedid »

ChuckMcM wrote: post #43

I am always amazed when otherwise intelligent people assert without data that the marginal cost of serving web traffic to scrapers/bots is zero. It is kind of like people who say "Why don't they put more fuel in the rocket so it can get all the way into orbit with just one stage?" It sounds great but it is a completely ignorant thing to say.

Seemingly, most of those people don't have a realistic concept of scale.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#54

Post by varenc »

TedDoesntTalk wrote: post #41
melony wrote: post #3

Earlier quoted context omitted.

The gold standard is residential IP. It is not cheap but its effectiveness is indisputable.

Not anymore. Now it’s mobile IP addresses.

This! Mobile IPs are far more lucrative. Many services will drop captchas and other anti-bot stuff for consumer mobile IPs. I recall Plaid at some point would run their bank scraping through mobile IPs.

This sketchy company lets mobile app developers monetize user base by letting other people pay $$ to route requests through random people’s mobile IPs: https://brightdata.com/

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#56

Post by spookthesunset »

oxymoron wrote: post #15
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

Yeah, I used to work for one of the major anti-bot vendors. Customers weren't clueless. Nobody buys these solutions because they're so much fun, it's a cost center and they monitor their ROI quite closely. Credit card charge backs, impact to infrastructure, extra incurred cost due to underlying api's (like in the Airline industry in particular) etc are all reasons why bot mitigation is a better option than nothing fo…

Not to mention a lot of these bots are after scamming the company’s own customers. Breaking into accounts to commit fraudulent activity, to reach out and “recruit” people into whatever scam they are trying to run.

Nobody wants to spend time trying to stop these bots. It is, however, a very necessary thing to do.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#57

Post by janmo »

curun1r wrote: post #17

There’s one technique that can be very useful in some circumstances that isn’t mentioned. Put simply, some sites try to block all bots except for those from the major search engines. They don’t want their content scraped, but they want the traffic that comes from search. In those cases, it’s often possible to scrape the search engines instead using specialized queries designed to get the content you want into the blu…

Or, you can spoof the google bot or Bing bot user agent and try to scrape the site that way.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#58

Post by EMIRELADERO »

Dah00n wrote: post #50
jonatron wrote: post #44

Earlier quoted context omitted.

Legitimate uses of scraping include price comparison, and finding copyright or trademark infringement.

>Legitimate uses of scraping include price comparison "Legitimate uses" is what the site operator says it is, nothing more nothing less. There are no laws that says you can scrape a site and circumvent their protection against doing so.

On the contrary, there are no laws that say you can't scrape a site. If it's available to the public internet, it's legally scrapable.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#59

Post by walrus01 »

Mister_Snuggles wrote: post #31
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#60

Post by spookthesunset »

walrus01 wrote: post #55

Google "residential proxies for sale" if you want to see the weird shady grey market for proxies when you need your traffic to come from things like cablemodem operator ASNs' DHCP pools

Wonder what fraction of that traffic is from p0wned IoT refrigerators, smoke detectors or WiFi enabled light bulbs… probably more than anybody cares to admit…
Post reply on HN