Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

positiveblue.substack.com

Post reply on HN
Print view View original

291–300 of 520 posts

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#291

Post by nitwit005 »

ctoth wrote: post #217

The web doesn't need attestation. It doesn't need signed agents. It doesn't need Cloudflare deciding who's a "real" user agent. It needs people to remember that "public" means PUBLIC and implement basic damn rate limiting if they can't handle the traffic. The web doesn't need to know if you're a human, a bot, or a dog. It just needs to serve bytes to whoever asks, within reasonable resource constraints. That's it. Th…

What you're proposing is that a lot of small websites should simply shut down, in the name of the open internet. The goals seem self contradictory.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#292

Post by JohnMakin »

Animats wrote: post #281

Are bots using a large number of IP addresses simultaneously, so they look like a DDOS attack? Or are they just making ordinary requests from a small number of addresses. If it's the latter, all you need is some kind of fair queuing so those requests compete with each other for access, not with other users.

Often it is rotating residential proxies. It is virtually impossible to mitigate this behavior from the IP level.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#293

Post by wvenable »

SamBam wrote: post #254
wvenable wrote: post #250

Earlier quoted context omitted.

You have a problem with badly behaved scrapers, not AI. I can't disagree with being against badly behaved scrapers. But this is neither a new problem or an interesting one from the idea of making information freely available to everyone, even rhinoceroses, assuming they are well behaved. Blocking bad actors is not the same thing as blocking AI.

But many people feel that the very act of incorporating your copyrighted words into their for-profit training set is itself the bad behavior. It's not about rate-limiting scrapers, it's letting them in the door in the first place.

Sure. But we're already talking about presumption of free and open here. I'm sure people are also reading my words and incorporating it into their own for-profit work. If I cared, I wouldn't make it free and open in the first place.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#294

Post by wvenable »

PhantomHour wrote: post #252
wvenable wrote: post #242

Earlier quoted context omitted.

Does allow bots to access my information prevent other people from accessing my information? No. If it did, you'd have a point and I would be against that. So many strange arguments are being made in this thread. Ultimately it is the users of AI (and am I one of them) that benefit from that service. I put out a lot of open code and I hope that people are able to make use of it however they can. If that's through AI,…

> Does allow bots to access my information prevent other people from accessing my information? No. Yes it does, that's the entire point. The flood of AI bots is so bad that (mainly older) servers are literally being overloaded and (newer servers) have their hosting costs spike so high that it's unaffordable to keep the website alive. I've had to pull websites offline because badly designed & ban-evading AI scraper bo…

That's a problem with scrapers, not with AI. I'm not sure why there are way more AI scraper bots now than there were search scraper bots back when that was the new thing. However that's still an issue of scapers and rate limiting and nothing to do with wanting or not wanting AI to read your free and open content.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#295

Post by Symbiote »

wvenable wrote: post #246
epc wrote: post #241

Earlier quoted context omitted.

Problem one is they do not honor the conventions of the web and abuse the sites. Problem two is they are taking content for free, distilling it into a product, and limiting access to that product.

Problem one is not specific to AI and not even about AI. Problem two is not anything new. Taking freely available content and distilling it into a product is something valuable and potentially worth paying for. People used to buy encyclopedias too. There are countless examples.

At present, problem one is almost entirely AI companies.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#296

Post by TheCycoONE »

Lammy wrote: post #288
AnthonyMouse wrote: post #273

Earlier quoted context omitted.

You can still do that, you have the caching reverse proxy at the edge of the network be the thing that terminates TLS.

Not really. At minimum you will break all of these sites on the HSTS preload list: https://source.chromium.org/chromium/chromium/src/+/main:net...

Public key pinning was rejected so you just need your proxy to also supply a certificate that's trusted by your clients.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#297

Post by wvenable »

bayindirh wrote: post #262
wvenable wrote: post #233

Earlier quoted context omitted.

> The problem is not AI bot scraping, per se, but "AI bot scraping while disregarding all licenses and ethical considerations". What licenses? Free and open web. Go crazy. What ethical considerations? Do I police how users use the information on my site? No. If they make a pipe bomb using an 6502 CPU using code taken from my website -- am I supposed to do something about that?

Creative Commons, GFDL, Unlicense, GPL/AGPL, MIT, WTFPL. Go crazy. I have the freedom to police how users use the information on my site. Yes. Real examples: My blog is BY-NC-SA and digital garden is GFDL. You can't take them, mangle and sell them. Especially, the blog. AI companies take these posts, and sell derivatives, without any references, consent or compensation. BY-NC-SA is complete opposite of what they do.…

Absolutely. If you want to put all kinds of copyright, license, and even payment restrictions on your content go ahead. And if AI companies or people abuse that, that's bad on them.

But I do think if you're serious about free and open information than why are you doing that in the first place? It's perfectly reasonable to be restrictive; I write both very open software and very closed software. But I see a lot of people want to straddle the line when it comes to AI without a rational argument.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#298

Post by wvenable »

danudey wrote: post #289
wvenable wrote: post #229

Earlier quoted context omitted.

Is that really the problem we are discussing? I've had people attack my server and bring it down. But that has nothing to do with being free and open to everyone. A top hacker news post could take my server.

Yes, because a top hacker news post takes your server down because a large number of actual humans are looking to gain actual value from your posts. Meanwhile, you stand to benefit from the HN discussion by learning new things and perspectives from the community. The AI bot assault, on the other hand, is one company (or a few companies) re-fetching the same data over and over again, constantly, in perpetuity, just in…

But then we get to use those AI tools.

The refrain here comes down not to "AI" but mostly to "the AI bot assault" which is a different thing. Sure lets have an discussion about badly behaved and overzealous web scrapers. As for credit, I've asked AI for it's references and gotten them. If my information is merely mushed into AI training model I'm not sure why I need credit. If you discuss this thread with your friends are you going to give me credit?

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#299

Post by thanatos_dem »

xdennis wrote: post #211
thanatos_dem wrote: post #88

Earlier quoted context omitted.

Allowlist is arguably fitting for a list of things which are allowed.

There are so many terms in software which are nonsensical (starting with "computer science") which could be fixed. The problem with changing whitelist to "allowlist" is that it implies that people who use whitelist are racists. You're not just virtue signaling (and confusing my spellchecker) but causing discord. It would be perfectly fine if people switched to "allowlist" because they think it's a better term, but th…

It’s an interesting time we live in.

I personally don’t assume people who use whitelist are racist, or those who allowlist are virtue signaling.

However, I certainly do assume that people who chastise others for using “whitelist” are virtue signaling, and those who deride people for using “allowlist” are racist.

Both are easily understood and I generally assume good intention from people. Just live and let live.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#300

Post by wvenable »

Symbiote wrote: post #295
wvenable wrote: post #246

Earlier quoted context omitted.

Problem one is not specific to AI and not even about AI. Problem two is not anything new. Taking freely available content and distilling it into a product is something valuable and potentially worth paying for. People used to buy encyclopedias too. There are countless examples.

At present, problem one is almost entirely AI companies.

And a few decades ago, it would have been search engine scrapers instead.
Post reply on HN