391–400 of 422 posts
Earlier quoted context omitted.
HN is surprisingly very very guilty of a whole lot of anti-user patterns and behaviour that other companies get regularly lamented. Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.
Poor accessibility Good.
Earlier quoted context omitted.
That's a completely different question, your claim was about parallelism.
Ah, I see what you're getting at. Yes. You can think of any given computer aa having a fixed amount of compute budget for these types of acceleration-resistant hashing algorithms. Let's say the scraper can perform 10,000 hashing operations per second total on their machine, and needs an average of 1,000 hashing operations to solve the PoW. It's a minor detail, but note that these PoW challenges non-deterministically…
Note that none of this assumes any sort of acceleration from parallelization (be it through GPUs or reusing work across servers) or precomputation relative to what a normal client does. Compute is just really cheap in dollars compared to the cost of having a user wait, and these companies _also_ have a lot of appetite for spending dollars compared to that of a normal user. As others have pointed out, the only reason why Anubis works (sort-of; not for everyone) right now is that it is uncommon enough, essentially “proof that you bothered to have your crawler run JavaScript at all”. It's a confusion measure.
Proof of work does not work.
Earlier quoted context omitted.
I think they also have to operate in countries that don't mind shady things like this.
Residential proxies aren't illegal in any country, within reason.
Earlier quoted context omitted.
Ah, I see what you're getting at. Yes. You can think of any given computer aa having a fixed amount of compute budget for these types of acceleration-resistant hashing algorithms. Let's say the scraper can perform 10,000 hashing operations per second total on their machine, and needs an average of 1,000 hashing operations to solve the PoW. It's a minor detail, but note that these PoW challenges non-deterministically…
So, in your example, each request needs 0.1 core-seconds of PoW. Is that right? Or, since you're saying 10000 hashes across the entire machine and 1000 needed, then on a typical 96-core server you need 9.6 core-seconds of PoW? The former means you pay about $0.0000005 per request at standard cloud rates; the latter means you pay about $0.00005 per request _and_ your site is totally unusable by legitimate clients. Bot…
Similarly, you have tarpits, which generate infinite mazes of garbage data, or even deliberately poisoning training data (should the scrapers be training LLMs) though this don't entirely eliminate the deleterious effects of scraping on the host's web server (more info: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-...).
If the premise was evaluating whether or not PoW would be a magic silver bullet that stops scrapers all by itself, then you are correct, it does not stop all scrapers. Scraping and anti-scraping is fundamentally a constantly evolving cat and mouse game that demands adaptability and punishes complacency from all participants trying not to lose.
The article at the end talks about how is very easy for arbitrary apps from app stores can install a residential proxy on your phone. 10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what. And both ios and android make it hard to deny apps network access. But really, this finally explains the hordes of real…
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.
It doesn't matter if page 42 of the Terms and Condition "clearly states it." This does not constitute informed consent.
Earlier quoted context omitted.
Noone complained we can't discriminate DC IPs, though to be fair some (imo bad) operators did just that. This is not even about preventing bots, which has perfectly legitimate usecases (eg. Internet Archive). This is about filtering out bad bots/actors who have no respect for your resources and will drain all of it causing bad experience for everyone. But because they know they don't respect robots.txt or even simple…
You could always try various counterattacks - returning a repeating compressed stream that decompresses to several terabytes, or returning one byte per second, or returning an endless list of hyperlinks to a honeypot, if the date is older than 200 years.
Thanks for the suggestions!
Earlier quoted context omitted.
The success of the ongoing Anubis rollout proves the opposite. People are used to slowly-loading websites - the rise of garbage SPAs has seen to that. Staring at a spinner for a second every once in a while is not an issue for genuine users. On the other hand, the additional CPU usage rises the compute cost of scraping by several orders of magnitude. If you don't have to scrape this specific website, you'd be stupid…
Scrapers generally aren't looking for random websites to scrape. They have a specific URL in mind. Only if the goal was DDoS would they not care which URL was accessed.
Earlier quoted context omitted.
Why is that good?
It's relatively difficult to enter and edit any significant amount of text on a phone, so phone-based discussions tend to be shallow.
Because I don’t buy it. There is no time limit for discussions. One can take as much time as they like to enter text on their phone. Also if they’re motivated by the discussion they may take the time and make the effort to write thoughtful replies.
Finally, there’s no reason that users visiting on desktop computers won’t make equally shallow remarks. Shallow online discussions have existed long before smartphones existed.
Sent from my iPhone.