Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

positiveblue.substack.com

Post reply on HN
Print view View original

371–380 of 520 posts

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#371

Post by 1gn15 »

ctoth wrote: post #198
TIPSIO wrote: post #20

Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…

"I want an open web!" "Okay, that means AI companies can train on your content." "Well, actually, we need some protections..." "So you want a closed web with access controls?" "No no no, I support openness! Can't we just have, like, ethical openness? Where everyone respects boundaries but there's no enforcement mechanism? Why are you making this so black and white?"

> “When we started the “free speech movement,” we had a bold new vision. No longer would dissenters’ views be silenced. With the government out of the business of policing the content of speech, robust debate and the marketplace of ideas would lead us toward truth and enlightenment. But it turned out that freedom of the press meant freedom for those who owned one. The wealthy and powerful dominated the channels of speech. The privileged had a megaphone and used free speech protections to immunize their own complacent or even hateful speech. Clearly, the time has come to denounce the naïve idealism of the past and offer a new movement, Speech 2.0, which will pay more attention to the political economy of media and aim at “free-ish” speech — the good stuff without the bad.”

https://openfuture.eu/paradox-of-open-responses/misunderesti...

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#372

Post by fsniper »

impure wrote: post #16

Well, if you have a better way to solve this that’s open I’m all ears. But what Cloudflare is doing is solving the real problem of AI bots. We’ve tried to solve this problem with IP blocking and user agents, but they do not work. And this is actually how other similar problems have been solved. Certificate authorities aren’t open and yet they work just fine. Attestation providers are also not open and they work just…

Are they? Until Let's Encrypt came along and democratise the CA scene, it was a hell hole. Web Security was depending on how deep your pockets are. One can argue that the same path is being laid in front us until a Let's Encrypt comes along and democratise it? And here as it's about attestation, how are we going to prevent gatekeeper's doing "selective attestations with arguable criteria"? How will we prevent political forces?

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#373

Post by tsimionescu »

giardini wrote: post #351
Twirrim wrote: post #328

Earlier quoted context omitted.

The only bot that bugs the crap out of me is Anthropic's one. They're the reason I set up a labyrinth using iocaine ( https://iocaine.madhouse-project.org/ ). Their bot was absurdly aggressive, particularly with retries. It's probably trivial in the whole scheme of things, but I love that anthropic spent months making about 10rps against my stupid blog, getting markov chain responses generated from the text of Moby D…

No wonder Anthropic isn't working well! The "Moby Dicked" explanation of the state of AI! But seriously, Why must someone search even a significant part of the public Internet to develop an AI? Is it believed that missing some text will cripple the AI? Isn't there some sort of "law of diminishing returns" where, once some percentage of coverage is reached, further scraping is not cost-effective?

On the contrary, AI training techniques require gigantic amounts of data to do anything, and there is no upper limit whatsoever - the more relevant data you have to train on, the better your model will be, period.

In fact, the biggest thing that is making it unlikely that LLM scaling will continue is that the current LLMs have already been trained on virtually every piece of human text we have access to today. So, without new training data (in large amounts), the only way they'll scale more is by new discoveries on how to train more efficiently - but there is no way to put a predictable timeline on that.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#374

Post by pixxel »

verdverm wrote: post #72
dlcarrier wrote: post #60

I use uncommon web browsers that don't leak a lot of information. To Cloudflare, I am indistingushable from a bot. Privacy cannot exist in an environment where the host gets to decide who access the web page. I'm okay with rate limiting or otherwise blocking activity that creates too much of a load, but trying to prevent automated access is impossible withou preventing access from real people.

The website owner has rights too. Are you arguing they cannot choose to implement such gatekeeping to keep their site operating in a financially viable manner?

[dead]

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#375

Post by 1gn15 »

jaredcwhite wrote: post #174

This article can easily be dismissed when hardly a moment in you see the headline "Agents Are Inevitable" I'm sorry, but the "agents" of "agentic AI" is completely different from the original purpose of the World-Wide Web which was to support user agents . User agents are used directly by users—aka browsers. API access came later, but even then it was often directed by user activity…and otherwise quite normally rate-…

Speak for yourself. This is just the semantic web: a web not built just for humans, but also for robots or any other types of agents that may wish to build upon the data. User agents never meant just web browsers, and operators blocking based on it necessitated hiding your identity.

Blocking bots is an absurd and unwinnable proposition, just like DRM; there's always the final, nuclear option of the analog hole, a literal video camera pointed at a monitor and using a keyboard and mouse.

If you really need to, deploy a proof of work shield that doesn't discriminate against user agents, just like what Onionsites do.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#376

Post by ymyms »

I agree with pretty much everything the author has said. I’ve been looking at the problem more on the enterprise side of things: how do you control what agents can and can’t do on a complex private network, let alone the internet.

I’ve actually just built an “identity token” using biscuit that you can delegate however you want after. So I can authenticate (to my service, but it could be federated or something just as well), get a token, then choose to create a delegated identity token from that for my agent. Then my agent could do the same for subagents.

In my system, you then have to exchange your identity token for an authorization token to do anything (single scope, single use).

For the internet, I’ve wondered about exchanging the identity token + a small payment (like a minuscule crypto amount) for an authorization token. Human users would barely spend anything. Bots crawling the web would spend a lot.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#377

Post by PaulRobinson »

lucb1e wrote: post #196
PaulRobinson wrote: post #98

Earlier quoted context omitted.

This doesn't scale to the general web, does it? I think invite-only might work to build communities, but you end up in the situation we're in today where people are buying/selling invites, and that's with treebans in place. I do fear the actions of the current bot landscape is going to lead to almost everything going behind auth walls though, and perhaps even paid auth walls.

I've been considering making this for the web. Why wouldn't it scale? Those selling invites would get banned soon enough if the people they distribute their invite to then send abusive traffic. Mystery shoppers can also make that a risky business if it's disallowed to sell invites (forcing them to be mostly free, such that the giver has nothing to gain from inviting someone who is willing to pay) One of the practical…

I don't think it scales, because I'm not sure it scales on private trackers already. I'm not deep into that space, but I think there's a lot of problems with it that will scale as adoption scales, particularly around policing the sale of invites - the hope would be it self-police through treebanning, but I'm not sure it does.

I think a sort of pseudo-anonymous auth system with backed in invites and treebans that website owners could easily adopt is interesting though. I'm not sure it's a business - for adoption reasons it likely needs to be a protocol - but it's an interesting idea, if it doesn't just turn into a huge admin headache for publishers.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#378

Post by jeroenhd »

rsync wrote: post #182
TIPSIO wrote: post #20

Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…

“But the reality is how can someone small protect their blog or content from AI training bots?” Why would you need to? If your inability to assemble basic HTML forces you to adopt enormous, bloated frameworks that require two full cores of a cpu to render your post… … or if you think your online missives are a step in the road to content creator riches … … then I suppose I see the problem. Otherwise there’s no proble…

I have had to disable at least one service because AI bots kept hitting it and it started impacting other stuff I was running that I am more interested in. Part of it was the CPU load on the database rendering dozens of 404s per second (which still required a database call), part of it was that the thumbnail images were being queried over and over again with seemingly different parameters for no reason.

I'm sure there are AI bots that are good and respect the websites they operate on. Most of them don't seem to, and I don't care enough about the AI bubble to support them.

When AI companies stop people from using them as cheap scrapers, I'll rethink my position. So far, there's no way to distinguish any good AI bot from a bad one.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#379

Post by lucumo »

RobSm wrote: post #363
BrenBarn wrote: post #247

Earlier quoted context omitted.

No, that is not true. It is only true if you just equate "AI training bots" with "people" on some kind of nominal basis without considering how they operate in practice. It is like saying "If your grocery store is open to the public, why is it not open to this herd of rhinoceroses?" Well, the reason is because rhinoceroses are simply not going to stroll up and down the aisles and head to the checkout line quietly wit…

You can always stop bots. Add login/password. But people want their content to be accessible to as large audience as possible, but at the same time they don't want that data to be accessible to the same audience via other channels. logic. Bots are not consuming your data - humans are. At the end of the day humans will eventually read it and take actions. For example chatgpt will mention your site, the user will visit…

> And no, nothing was different before 2022. Just look at google, the largest bot scraping network in the world. Since 1996.

I'm sorry, but this statement shows you have no recent experience with these crawlernets.

Google, from the beginning, has done their best to work with server owners. They respect robots.txt. I think they were the first to implement Crawl-Delay. They crawl based on how often things change anyway. They have an additional safeguard that when they notice a slowdown in your responses, they back off.

Compare this with Anthropic. On their website they say they follow robots.txt and Crawl-Delay. I have an explicit ban on Claudebot in there and a Crawl-Delay for everyone else. It ignores both. I send an email to them about this, and their answer didn't address the discrepancy between the docs and the behaviour. They just said they'll add me to their internal whitelist and that I should've sent 429s when they were going too fast. (Fuck off, how about you follow your public documentation?)

That's just my experience, but if you Google around you'll find that Anthropic is notorious for ignoring robots.txt.

And still, Claudebot is one of the better behaved bots. At least they identify themselves, have a support email they respond to, and use identifiable IP-addresses.

A few weeks ago I spend four days figuring out why I had 20x the traffic I normally have (which maxed out the server; causing user complaints). Turns out there are parties that crawl using millions of (residential) IPs and identify themselves as normal browsers. Only 1 or 2 connections per IP at the time. Randomization of identifying properties. Even Anthropics 429 solution wouldn't have worked there.

I managed to find a minor identifying property in some of the requests that wasn't catching too many real users. I used that to start firewalling IPs on sight and then their own randomization caused every IP to fall into the trap in the end. But it took days.

In the end I had to firewall nearly 3 million non-consecutive IP addresses.

So no, Google in 1996 or 2006 or 2016 is not the same as the modern DDoSing crawlernet.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#380

Post by jeroenhd »

impure wrote: post #16

Well, if you have a better way to solve this that’s open I’m all ears. But what Cloudflare is doing is solving the real problem of AI bots. We’ve tried to solve this problem with IP blocking and user agents, but they do not work. And this is actually how other similar problems have been solved. Certificate authorities aren’t open and yet they work just fine. Attestation providers are also not open and they work just…

I'm not sure if things are as fine as you say they are. Certificate authorities were practically unheard of outside of corporate websites (and even then mostly restricted to login pages) until Let's Encrypt normalized HTTPS. Without the openness of Let's Encrypt, we'd still be sharing our browser history and search queries with our ISPs for data mining. Attestation providers have so far refused to revoke attestation for known-vulnerable devices (because customers needing to replace thousands of devices would be an unacceptable business decision), making the entire market rather useless.

That said, what I am missing from these articles is an actual solution. Obviously we don't want Cloudflare from becoming an internet gatekeeper. It's a bad solution. But: it's a bad solution to an even worse problem.

Alternatives do exist, even decentralised ones, in the form of remote attestation ("can't access this website without secure boot and a TPM and a known-good operating system"), paying for every single visit or for subscriptions to every site you visit (which leads to centralisation because nobody wants a subscription to just your blog), or self-hosted firewalls like Anubis that mostly rely on AI abuse being the result of lazy or cheap parties.

People drinking the AI Kool-Aid will tell you to just ignore the problem, pay for the extra costs, and scale up your servers, because it's *the future*, but ignoring problems is exactly why Cloudflare still exists. If ISPs hadn't ignored spoofing, DDoS attacks, botnets within their network, """residential proxies""", and other such malicious acts, Cloudflare would've been an Akamai competitor rather than a middle man to most of the internet.

Post reply on HN