Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

positiveblue.substack.com

Post reply on HN
Print view View original

231–240 of 520 posts

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#231

Post by lucb1e »

remich wrote: post #209
lucb1e wrote: post #183

Earlier quoted context omitted.

The first article of our constitution says people shall be treated equally in equal situations. I presume that most countries have similar clauses but, beyond legalese, it's also simply in line with my ethics to treat everyone equally There are people behind those connection requests. I don't try to guess on my server who is a bot and who is not; I'll make mistakes and probably bias against people who use uncommon se…

If by "our constitution" you mean the U.S. Constitution then no, it says nothing of the sort. The first article of the U.S. Constitution concerns the organization of the legislative branch. You may be referencing the Equal Protection and Due Process clauses, in the Fifth and Fourteenth amendments, but neither of those applies in this situation either since there are no laws or governmental actions at issue here, and…

You can assume it's the USA and that I'm just dead wrong, but the third word of my profile specifies where I'm from and you'd find that this Dutch constitution matches the comment's contents

Equal protection is indeed not the same as equal treatment. No, it really does say that everyone shall be treated equally so long as the circumstances are equal (gelijke behandeling in gelijke gevallen)

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#232

Post by [deleted] »

bob1029 wrote: post #21
jjangkke wrote: post #3

I would love to get off Cloudflare but there are no real good alternatives

Writing backends that can actually handle public traffic and using authentication for expensive resources are fantastic alternatives. Also, cheaply rate limiting malicious web clients should be something that is trivial to accomplish with competent web tooling (i.e., on your own servers). If this seems out of scope or infeasible, you might be using the wrong tools for the job.

[deleted]

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#233

Post by wvenable »

bayindirh wrote: post #187
wilson090 wrote: post #176

Earlier quoted context omitted.

How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.

The problem is not AI bot scraping, per se, but "AI bot scraping while disregarding all licenses and ethical considerations". Freedom, the word, while implies no boundaries, is always bound by ethics, mutual respect and "do no harm" principle. The moment you trip either one of these wires and break them, the mechanisms to counter it becomes active. Then we cry "but, freedom?!". Freedom also contains the consequences…

> The problem is not AI bot scraping, per se, but "AI bot scraping while disregarding all licenses and ethical considerations".

What licenses? Free and open web. Go crazy. What ethical considerations? Do I police how users use the information on my site? No. If they make a pipe bomb using an 6502 CPU using code taken from my website -- am I supposed to do something about that?

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#234

Post by calmbonsai »

While I concur with the effective tech, I don't think this is something that's a net win for society.

Just because you can, doesn't mean you should and I don't feel any one entity (private or public) should be an arbiter on these matters.

This is something that can, and should, be negotiated at the "last virtual mile".

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#235

Post by lucb1e »

Gud wrote: post #225
lucb1e wrote: post #214

Earlier quoted context omitted.

If you're handling billions of requests per second, you're not a self hoster. That's a commercial service with a dedicated team to handle traffic around the clock. Most ISPs probably don't even operate lines that big To put that in perspective, even if they're sending empty TCP packets, "several billion" pps is 200 to 1800 gigabits of traffic, depending on what you mean by that. Add a cookieless HTTP payload and you'…

To be fair, he did say per minute :-)

Oh, whoops. Divide everything by 60, quick!

That does make it a bit less ludicrous even if I think the conclusion of my response still applies

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#236

Post by wvenable »

epc wrote: post #221
wilson090 wrote: post #176

Earlier quoted context omitted.

How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.

Do the AI training bots provide free access to the distillation of the content they drain from my site repeatedly? Don't they want a free and open web? I don’t feel a particular need to subsidize multi–billion even trillion dollar corporations with my content, bandwidth, and server costs since their genius vibe coded bots apparently don’t know how to use modified-GETs or caching, let alone parse and respect robots.tx…

Is the problem they exist or the problem they are badly accessing your site? Because there are two conflating issues here. If humans or robots are causing you issues, as both can do, that's bad. But that has nothing to do with AI in particular.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#237

Post by jopsen »

gausswho wrote: post #71
TIPSIO wrote: post #20

Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…

What we need is some legal teeth behind robots.txt. It won't stop everyone, but Big Corp would be a tasty target for lawsuits.

I have the feeling that it's the small players that cause problems.

Dumb bots that don't respect robot.txt or nofollow are the ones trying all combinations of the filters available in your search options and requesting all pages for each such combination.

The number of search pages can easily be exponential in the number of filters you offer.

Bots walking around in these traps, do it because they are dumb. But even a small degenerate bot can send more requests than 1M MAUs.

At least that's my impression of the problem we're sometimes facing.

Signed agents seems like a horrific solution. And many serving the traffic is just better.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#238

Post by ctoth »

johncolanduoni wrote: post #227
ctoth wrote: post #217

The web doesn't need attestation. It doesn't need signed agents. It doesn't need Cloudflare deciding who's a "real" user agent. It needs people to remember that "public" means PUBLIC and implement basic damn rate limiting if they can't handle the traffic. The web doesn't need to know if you're a human, a bot, or a dog. It just needs to serve bytes to whoever asks, within reasonable resource constraints. That's it. Th…

Basic damn rate limiting is pretty damn exploitable. Even ignoring botnets (which is impossible), usefully rate limiting IPv6 is anything but basic. If you just pick some prefix from /48 to /64 to key your rate limits on, you'll either be exploitable by IPs from providers that hand out /48s like candy or you'll bucket a ton of mobile users together for a single rate limit.

You make unauthenticated requests cheap enough that you don't care about volume. Reserve rate limiting for authenticated users where you have real identity. The open web survives by being genuinely free to serve, not by trying to guess who's "real."

A basic Varnish setup should get you most of the way there, no agent signing required!

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#239

Post by BobaFloutist »

wilson090 wrote: post #176
BrenBarn wrote: post #175

Earlier quoted context omitted.

I think that was the point. Everyone loves the dream, but the reality is different.

How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.

> information is free and available for anyone.

Bots aren't people.

You can want public water fountains without wanting a company attaching a hose to the base to siphon municipal water for corporate use, rendering them unusable for everyone else.

You can want free libraries without companies using their employees' library cards to systematically check out all the books at all times so they don't need to wait if they want to reference one.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#240

Post by zzo38computer »

With what they say about authorization, I think X.509 would help. (Although central certificate authorities are often used with X.509, it does not have to be that way; the service you are operating can issue the certificate to you instead, or they can accept a self-signed certificate which is associated with you the first time it is used to create an account on their service.)

You can use the admin certificate issued to you, to issue a certificate to the agent which will contain an extension limiting what it can be used for (and might also expire in a few hours, and also might be revoked later). This certificate can be used to issue an even more restricted certificate to sub-agents.

This is already possible (and would be better than the "fine-grained personal access tokens" that GitHub uses), but does not seem to be commonly implemented. It also improves security in other ways.

So, it can be done in such a way that Cloudflare does not need to issue authorization to you, or necessarily to be involved at all. Google does not need to be involved either.

However, that is only for things where would should normally require authorization to do anyways. Reading public data is not something that should requires authorization to do; the problem with this is excessive scraping (there seems to be too many LLM scraping and others which is too excessive) and excessive blocking (e.g. someone using a different web browser, or curl to download one file, or even someone using a common browser and configuration but something strange unexpected happens, etc); the above is something unrelated to that, so certificates and stuff like that does not help, because it solves a different problem.

Post reply on HN