Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

61–70 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#61

Post by walrus01 »

TedDoesntTalk wrote: post #41
melony wrote: post #3

Earlier quoted context omitted.

The gold standard is residential IP. It is not cheap but its effectiveness is indisputable.

Not anymore. Now it’s mobile IP addresses.

One of the reasons for this is that the vast majority of the time, mobile LTE data users are behind cgnat for ipv4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point.

As a scraper operator on a mobile data connection all you need is a new useragent and browser fingerprint, there's no easy way for a scraper-blocker-operator to tell that you're not a totally new person.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#62

Post by dragonwriter »

EMIRELADERO wrote: post #58
Dah00n wrote: post #50

Earlier quoted context omitted.

>Legitimate uses of scraping include price comparison "Legitimate uses" is what the site operator says it is, nothing more nothing less. There are no laws that says you can scrape a site and circumvent their protection against doing so.

On the contrary, there are no laws that say you can't scrape a site. If it's available to the public internet, it's legally scrapable.

> On the contrary, there are no laws that say you can't scrape a site.

You are both wrong: copyright law both says you can't (in some cases for some uses) and that you can (under implicit license, fair use, and other rules) in others.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#63

Post by walrus01 »

spookthesunset wrote: post #60
walrus01 wrote: post #55

Google "residential proxies for sale" if you want to see the weird shady grey market for proxies when you need your traffic to come from things like cablemodem operator ASNs' DHCP pools

Wonder what fraction of that traffic is from p0wned IoT refrigerators, smoke detectors or WiFi enabled light bulbs… probably more than anybody cares to admit…

Also a lot of people who've been tricked into installing malware on their windows PCs, from shady "VPN" operators and other

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#64

Post by sparkling »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

> Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks.

I disagree. Obviously there is no way to 100% stop scraping, but a for a rather small amount of $ you can implement some measures that make it harder. Services like https://focsec.com/ offer ways to detect web scrapers using proxy/VPNs (one of the most common techniques) for little money.

> Nobody pays those vendors $10m/year to frustrate web crawler enthusiasts, they do it to stop credential stuffing.

Keep in mind that they may be legally or contractually forced to do this. Think of Netflix who are investing heavily into their Anti-VPN capabilities, most likely because they have contracts with content publishers & studios that force them to do so.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#65

Post by marginalia_nu »

EMIRELADERO wrote: post #58
Dah00n wrote: post #50

Earlier quoted context omitted.

>Legitimate uses of scraping include price comparison "Legitimate uses" is what the site operator says it is, nothing more nothing less. There are no laws that says you can scrape a site and circumvent their protection against doing so.

On the contrary, there are no laws that say you can't scrape a site. If it's available to the public internet, it's legally scrapable.

There are laws against unauthorized computer access.

This is a scenario where you have a server explicitly saying "Stop! You are not permitted to access this computer!", and yet you persist in circumventing that by hiding your identity and accessing it anyway. Those are some murky waters.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#66

Post by catlifeonmars »

folmar wrote: post #28
devit wrote: post #11

Earlier quoted context omitted.

If users using weak/reused passwords is your problem, just don't let users choose a password (generate it for them), or don't use passwords at all (send link by e-mail that adds a cookie), or use oauth login.

Link-only login is the most underused security option, even more so for low-profile sites that need a minimal user account but do not really need full-on security.

I’m curious. What is link-only login?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#67

Post by sparkling »

melony wrote: post #3
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

The gold standard is residential IP. It is not cheap but its effectiveness is indisputable.

There are services that detect residential IPs being used for scraping nowadays. Plus there are other ways of detecting scraping: browser fingerprinting, aggressive rate-limiting and CAPTCHAs etc.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#68

Post by walrus01 »

Gigachad wrote: post #8
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

2FA should be a requirement on everything now. And if your site can't for some reason or you don't want to deal with it, then limit your site to external login providers only. 2FA, especially app based, has been proven to work really really well.

How do you propose to implement two-factor authentication, on something like the public facing homepage of an airline ticket price search website, where if you make people "sign in with google" or whatever, a sizeable proportion won't do it and will just go to the competition?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#70

Post by mdoms »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

If it was just about credential stuffing they would only put limits on POST requests.
Post reply on HN