581–590 of 668 posts
Earlier quoted context omitted.
The article is from 2024. Is this still happening?
Do we have any evidence they started complying? If not, we can conclude they did not, until such evidence shows up.
Earlier quoted context omitted.
>The real frustrating part is that Cloudflare's "definition" of suspicious keeps changing and expanding. That's... exactly expected? It's a cat and mouse game. People running botnets or AI scrapers aren't diligently setting the evil bit on their packets.
So the stable state here is all humans eventually being locked out? (Bots are getting better every day; I doubt the same is true for all humans, including those with weird browsers or networks unwilling to install some dystopian Cloudflare "Internet passport".) But hey, at least some bots are also not making it past Cloudflare!
i.e. iCloud private relay is the future
Earlier quoted context omitted.
You cannot know what verifications they use. I could argue the disabled textbox is some sort part of the verification process. Humans will click on it while bots won't.
Seems like a trivially simple verification to defeat.
Earlier quoted context omitted.
Do we have any evidence they started complying? If not, we can conclude they did not, until such evidence shows up.
I’m genuinely curious to know whether there was a change in behavior especially after OpenAI informed about how to prevent scraping (robot.txt, etc.).
I just don't understand why bot owners can't just run a complete windows 11 VM running Google Chrome complete with graphics acceleration. You can probably run 50 of those simultaneously if you use memory page deduplication, and with a decent CPU+GPU you ought to be able to render 50 pages a second. That's 1 cent per thousand page loads on AWS. Damn cheap.
Earlier quoted context omitted.
How does the Tor network counter abuse? Like, say you're hosting a service on the Tor network, what does the Tor network offer if anything to defend against e.g. DDoS attacks?
It's a solution for users because you can't afford to demand ID from your users (such as an IP address) if all your users quit when you do that.
All it would solve then is laundering Tor traffic from being probably malicious to being reputationally ambiguous. Though for a within-network service, that's probably assumed anyways - hard to run a Tor service if you assume all Tor users are malicious, that would be nonsensical.
Earlier quoted context omitted.
Manifest destiny for me, border enforcement for thee.
This kind of flawed thinking again. Like the natives didn't fight and lose wars against the manifest destiny types.
Earlier quoted context omitted.
Lets not try to qualify the wrongs by picking a metric and evaluating just one side of it. A static website owner could be running with a very small budget and the scraping from bots can bring down their business too. The chances of a static website owner burning through their own life savings are probably higher.
If you're truly running a static site, you can run it for free, no matter how much traffic you're getting. Github pages is one way, but there are other platforms offering similar services. Static content just isn't that expensive to host. THe troubles start when you're actually running something dynamic that pretends to be static, like Wordpress or Mediawiki. You can still reduce costs significantly with CDNs / cachi…
https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-th...
Earlier quoted context omitted.
That still doesn’t explain why you can’t even start typing until that check proceeds. You could condition the outbound request from being processed until that’s the case. But preventing from typing seems like it’s just worse UX and the problem will fail to appear in any metrics you can track because you have no way of measuring “how quickly would the user have submitted their request without all this other stuff in t…
Sounds like anti-distillation to me. But, know what? Meh.
Earlier quoted context omitted.
Hi Nick, first of all, very cool of you to respond here instead of letting us all sit in the dark. I think that's what makes HN special. That said, is it not a little bit weird that you want to protect yourself from scraping and bots, when your entire company, product, revenue, and your employment, depends on the fact that OpenAI can bot and scrape literally every part of the internet? So your moat is non-hydrated re…
Don't beat up an engineer for decisions made by company leadership. It's really inappropriate.