Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

positiveblue.substack.com

Post reply on HN
Print view View original

251–260 of 520 posts

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#251

Post by Symbiote »

bob1029 wrote: post #21
jjangkke wrote: post #3

I would love to get off Cloudflare but there are no real good alternatives

Writing backends that can actually handle public traffic and using authentication for expensive resources are fantastic alternatives. Also, cheaply rate limiting malicious web clients should be something that is trivial to accomplish with competent web tooling (i.e., on your own servers). If this seems out of scope or infeasible, you might be using the wrong tools for the job.

If it were this easy, we wouldn't have had about 10 HN posts on the topic in the last few months.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#252

Post by PhantomHour »

wvenable wrote: post #242
BobaFloutist wrote: post #239

Earlier quoted context omitted.

> information is free and available for anyone. Bots aren't people. You can want public water fountains without wanting a company attaching a hose to the base to siphon municipal water for corporate use, rendering them unusable for everyone else. You can want free libraries without companies using their employees' library cards to systematically check out all the books at all times so they don't need to wait if they…

Does allow bots to access my information prevent other people from accessing my information? No. If it did, you'd have a point and I would be against that. So many strange arguments are being made in this thread. Ultimately it is the users of AI (and am I one of them) that benefit from that service. I put out a lot of open code and I hope that people are able to make use of it however they can. If that's through AI,…

> Does allow bots to access my information prevent other people from accessing my information? No.

Yes it does, that's the entire point.

The flood of AI bots is so bad that (mainly older) servers are literally being overloaded and (newer servers) have their hosting costs spike so high that it's unaffordable to keep the website alive.

I've had to pull websites offline because badly designed & ban-evading AI scraper bots would run up the bandwidth into the TENS OF TERABYTES, EACH. Downloading the same jpegs every 2-3 minutes into perpetuity. Evidently all that vibe coding isn't doing much good at Anthropic and Perplexity.

Even with my very cheap transfer racks up $50-$100/mo in additional costs. If I wanted to use any kind of fanciful "app" hosting it'd be thousands.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#253

Post by JohnMakin »

> The same is true online. A cryptographic signature that claims “I am acting on behalf of X” means nothing unless it is tied to something real, like a verifiable infrastructure or a range of IPs. Without that, I can simply hand the passport to another agent, and they can act as if they were me. The passport becomes nothing more than a token anyone can pass around.

how does this person think jwt’s work?

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#254

Post by SamBam »

wvenable wrote: post #250
BrenBarn wrote: post #247

Earlier quoted context omitted.

No, that is not true. It is only true if you just equate "AI training bots" with "people" on some kind of nominal basis without considering how they operate in practice. It is like saying "If your grocery store is open to the public, why is it not open to this herd of rhinoceroses?" Well, the reason is because rhinoceroses are simply not going to stroll up and down the aisles and head to the checkout line quietly wit…

You have a problem with badly behaved scrapers, not AI. I can't disagree with being against badly behaved scrapers. But this is neither a new problem or an interesting one from the idea of making information freely available to everyone, even rhinoceroses, assuming they are well behaved. Blocking bad actors is not the same thing as blocking AI.

But many people feel that the very act of incorporating your copyrighted words into their for-profit training set is itself the bad behavior. It's not about rate-limiting scrapers, it's letting them in the door in the first place.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#255

Post by PaulHoule »

SamBam wrote: post #254
wvenable wrote: post #250

Earlier quoted context omitted.

You have a problem with badly behaved scrapers, not AI. I can't disagree with being against badly behaved scrapers. But this is neither a new problem or an interesting one from the idea of making information freely available to everyone, even rhinoceroses, assuming they are well behaved. Blocking bad actors is not the same thing as blocking AI.

But many people feel that the very act of incorporating your copyrighted words into their for-profit training set is itself the bad behavior. It's not about rate-limiting scrapers, it's letting them in the door in the first place.

Why was it OK for Google to incorporate their words into a for-profit search index which has increasingly sucked all the profit out of the system?

My Ithaca friends on Facebook complain incessantly about the very existence of AI to the extent that I would not want to say I ask Copilot how to use Windows Narrator or Junie where the CSS that makes this text bold or sometimes have Photoshop draw an extra row of bricks in a photograph for me.

The same people seem to have no problem with Facebook using their words for all things Facebook uses them for, however.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#256

Post by alemanek »

wvenable wrote: post #172
TIPSIO wrote: post #20

Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…

> Everyone loves the dream of a free for all and open web... But the reality is how can someone small protect their blog or content from AI training bots? Aren't these statements entirely in conflict? You either have a free for all open web or you don't. Blocking AI training bots is not free and open for all.

And? Paying Cloudflare or someone else to block bad actors is required these days unless you have the scale and expertise to do it yourself.

Why is outsourcing this to Cloudflare bad and doing it yourself ok? Am I allowed to buy a license to a rate limiter or do I need to code my own? Am I allowed to use a firewall or is blocking people from probing my server not free enough?

Why are bots or any other user entitled to unlimited visits to my website? The entitlement is kind of unreal at this point

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#257

Post by pavel_lishin »

Loughla wrote: post #218
pton_xd wrote: post #197

Earlier quoted context omitted.

> If you don't want AI bots reading information on the web, you don't actually want a free and open web. This is such a bad faith argument. We want a town center for the whole community to enjoy! What, you don't like those people shooting up drugs over there? But they're enjoying it too, this is what you wanted right? They're not harming you by doing their drugs. Everyone is enjoying it!

Set aside that there's a pretty big difference between AI scraping and illegal drug usage. If the person using illegal drugs is on no way harming anyone but themselves and not being a nuisance, then yeah, I can get behind that. Put whatever you want in your body, just don't let it negatively impact anyone around you. Seems reasonable?

> just don't let it negatively impact anyone around you.

Exactly! Which is why we don't want AI bots siphoning our bandwidth & processing power.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#258

Post by PaulHoule »

lucb1e wrote: post #214
victorbjorklund wrote: post #137

Earlier quoted context omitted.

I self-host lots of stuff. But yes it is more pain to host a WAF that can handle billions of request per minute. Even harder to do it for free like Cloudflare. And in the end the end result for the user is exactly the same if you use a self-hosted WAF or let someone else host it for you.

If you're handling billions of requests per second, you're not a self hoster. That's a commercial service with a dedicated team to handle traffic around the clock. Most ISPs probably don't even operate lines that big To put that in perspective, even if they're sending empty TCP packets, "several billion" pps is 200 to 1800 gigabits of traffic, depending on what you mean by that. Add a cookieless HTTP payload and you'…

If somebody decides they hate you, your site that could handle, say, 100,000 legitimate requests per day could suddenly get billions of illegitimate requests.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#259

Post by IshKebab »

krapp wrote: post #184
IshKebab wrote: post #162

Earlier quoted context omitted.

It's called a whitelist. A perfectly good word that isn't racist and one that normal people are quite happy to use. As far as I can tell the allow/blocklist craze hasn't made it out of the software world.

Both whitelist and allowlist are equally normal and good. It's weird that people will claim that "politics" have no place in software while insisting that there is one and only one term "normal" people should use because the politics of the people who object to it are bad and wrong.

> Both whitelist and allowlist are equally normal and good.

Good is debatable, but normal?? No, obviously not. One is a word that has been around for over a hundred years and is understood by everyone that speaks English; the other is like 4 years old and only used by some software nerds. 90% of normal people would not know what it means.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#260

Post by Lammy »

ctoth wrote: post #238
johncolanduoni wrote: post #227

Earlier quoted context omitted.

Basic damn rate limiting is pretty damn exploitable. Even ignoring botnets (which is impossible), usefully rate limiting IPv6 is anything but basic. If you just pick some prefix from /48 to /64 to key your rate limits on, you'll either be exploitable by IPs from providers that hand out /48s like candy or you'll bucket a ton of mobile users together for a single rate limit.

You make unauthenticated requests cheap enough that you don't care about volume. Reserve rate limiting for authenticated users where you have real identity. The open web survives by being genuinely free to serve, not by trying to guess who's "real." A basic Varnish setup should get you most of the way there, no agent signing required!

> You make unauthenticated requests cheap enough that you don't care about volume.

In the days before mandatory TLS it was so easy to set up a Squid proxy on the edge of my network and cache every plain-HTTP resource for as long as I want.

Like yeah, yeah, sure, it sucked that ISPs could inject trackers and stuff into page contents, but I'm starting to think the downsides of mandatory TLS outweigh the upsides. We made the web more Secure at the cost of making it less Private. We got Google Analytics and all the other spyware running over TLS and simultaneously made it that much harder for any normal person to host anything online.

Post reply on HN