251–260 of 422 posts
Earlier quoted context omitted.
I feel less bad hammering firebase in a "while True:" loop vs hitting HN's servers.
12,000 times less bad? https://news.ycombinator.com/item?id=48868910 >
You're right to point out that if you're trying to get the contents of dead objects the API is of no use though.
Earlier quoted context omitted.
Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.
I doubt they are doing it. You just have to get on a VPN to and see yourself being flooded with captchas despite browsing the web like a normal human and solving dozens of captchas along the way.
Earlier quoted context omitted.
It's not about traffic identification at all, but rather a hashing algorithm that is deliberately resistant to parallelization and GPU/ASIC acceleration, which shrinks the gap in solving speed between the fastest systems (i.e. datacenter-class compute resources) and typical systems (e.g. the CPU in your smartphone or laptop).
Uh, is it resistant to parallelization across multiple sites? Because that's the situation for the scrapers. They're not trying to solve a single PoW challenge across many cores.
In effect, if the customer (the entity paying for and using the proxies) wants to solve PoW challenges through those connections, it is indeed the customer who must pay that compute cost, not the compromised devices.
Note that this is the case for a majority of, but not all, residential proxy networks, which often are built through quasi-voluntary distribution channels, including SDKs included in otherwise legitimate mobile applications distributed through Apple's App Store and Google Play.
These distribution channels tend to be categorically unavailable (or at least unreliable) for true RAT-style malware that enables remote operators to dynamically assign arbitrary computational workloads to client devices.
This isn't to say that true botnets built with actual malware delivered through either software exploits, phishing attacks, or watering hole attacks don't also perform as residential proxy networks, but such categories are a relatively small subset of all residential proxy networks, and there are much higher ROI malicious activities to be performed on these devices rather than serving as relatively mundane traffic networks for scraping.
Earlier quoted context omitted.
This seems nonsensical. Care to elaborate?
If Google determines you're an undesirable user, doing the captcha is just an exercise to waste your time.
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…
Or you can go full Reddit and just block anything that seems even remotely suspicious. Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too. Using VPN? Blocked. Your iPhone is too old, blocked. Your screen brightness too low? Believe or not, blocked.
Earlier quoted context omitted.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…
> a badly configured scraping setup? Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s. Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmuck…
Earlier quoted context omitted.
PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
If you pop my machine and use it to route 100 MBit/s, I might not notice for months. If I hear the fan spinning at night, you're probably getting caught immediately. If you pop my mom's TV box and use it to route data within the connection's capabilities, you're getting away with it. If you consume a little bit of resources, still. If you consume enough to be useful for these kind of challenges, chances are her TV pl…
video decoding is hardware accelerated, and there's probably enough excess compute to be able to do some sort of PoW challenge. Besides, unlike humans, bots aren't in a hurry, so they can spread out the work across a long time to minimize disruption.
Earlier quoted context omitted.
PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
If you pop my machine and use it to route 100 MBit/s, I might not notice for months. If I hear the fan spinning at night, you're probably getting caught immediately. If you pop my mom's TV box and use it to route data within the connection's capabilities, you're getting away with it. If you consume a little bit of resources, still. If you consume enough to be useful for these kind of challenges, chances are her TV pl…
Earlier quoted context omitted.
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
Cloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
Earlier quoted context omitted.
> Unfortunately whatever HN is using routinely blocks my login with "Sorry." I believe that's the HN application itself, not a WAF in front of it.
HN is surprisingly very very guilty of a whole lot of anti-user patterns and behaviour that other companies get regularly lamented. Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.