Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

positiveblue.substack.com

Post reply on HN
Print view View original

221–230 of 520 posts

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#221

Post by epc »

wilson090 wrote: post #176
BrenBarn wrote: post #175

Earlier quoted context omitted.

I think that was the point. Everyone loves the dream, but the reality is different.

How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.

Do the AI training bots provide free access to the distillation of the content they drain from my site repeatedly? Don't they want a free and open web?

I don’t feel a particular need to subsidize multi–billion even trillion dollar corporations with my content, bandwidth, and server costs since their genius vibe coded bots apparently don’t know how to use modified-GETs or caching, let alone parse and respect robots.txt.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#222

Post by Bender »

IshKebab wrote: post #83

[flagged]

Off topic but are people ever going to give up on this nonsense?

/shrug. FoxReplace [1] is a simple way to remove compelled speech from the internet for the client. People do not even realize when they say "Fart", I see "Fart". No idea if Chrome has an addon like this.

[1] - https://addons.mozilla.org/en-US/firefox/addon/foxreplace/

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#223

Post by zzo38computer »

krapp wrote: post #184
IshKebab wrote: post #162

Earlier quoted context omitted.

It's called a whitelist. A perfectly good word that isn't racist and one that normal people are quite happy to use. As far as I can tell the allow/blocklist craze hasn't made it out of the software world.

Both whitelist and allowlist are equally normal and good. It's weird that people will claim that "politics" have no place in software while insisting that there is one and only one term "normal" people should use because the politics of the people who object to it are bad and wrong.

I agree that both words are good, but there is a difference.

Whitelist means that anything explicitly listed (in the "whitelist" or "allow list") is allowed (or included, etc) and other stuff is disallowed (or excluded) by default (although in some cases, a program (or something else) might ask instead of forcibly blocking access). It is a compound word; you should not use a space or hyphen. (Using two words "white list" may be appropriate when you are refering to colours, e.g. the white list includes the list of whatever documents are to be copied on white paper, or "white list" might mean the list that is printed on white paper.)

Allow list (I do not like the compound word; I think they should be separated and it looks better that way) is the list of what is allowed. (So, normally, this would mean that other stuff is not allowed, so it is still whitelisting.)

In situations where colours would be involved and using words such as "whitelist" would be confusing, such words should be avoided, in order to avoid confusion.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#224

Post by ianbutler »

Symbiote wrote: post #200
qwerty456127 wrote: post #151

Earlier quoted context omitted.

What we need is stop fighting robots and start welcoming and helping them. I se zero reasons to oppose robots visiting any website I would build. The only purpose I ever tried disallowed robots for was preventing search engines from indexing incomplete versions or going the paths which really make no sense for them to go. Now I think we should write separate instructions for different kinds of robots: a search engine…

> I see zero reasons to oppose robots visiting any website I would build. > preventing search engines from indexing incomplete versions or going the paths which really make no sense for them to go. What will you do when the bots ignore your instructions, and send a million requests a day to these URLs from half a million different IP addresses?

Let my site go down and then restart my server a few hours later. I'm a dude with a blog I'm not making uptime guarantees. I think you're overestimating the harm and how often this happens.

Misbehaving scrapers have been a problem for years not just from AI. I've written posts on how to properly handle scraping and the legal grey area it puts you in and how to be a responsible one. If companies don't want to be responsible the solution isn't abdicate an open web. It's make better law and enforcement of said law.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#225

Post by Gud »

lucb1e wrote: post #214
victorbjorklund wrote: post #137

Earlier quoted context omitted.

I self-host lots of stuff. But yes it is more pain to host a WAF that can handle billions of request per minute. Even harder to do it for free like Cloudflare. And in the end the end result for the user is exactly the same if you use a self-hosted WAF or let someone else host it for you.

If you're handling billions of requests per second, you're not a self hoster. That's a commercial service with a dedicated team to handle traffic around the clock. Most ISPs probably don't even operate lines that big To put that in perspective, even if they're sending empty TCP packets, "several billion" pps is 200 to 1800 gigabits of traffic, depending on what you mean by that. Add a cookieless HTTP payload and you'…

To be fair, he did say per minute :-)

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#226

Post by zzo38computer »

xdennis wrote: post #211
thanatos_dem wrote: post #88

Earlier quoted context omitted.

Allowlist is arguably fitting for a list of things which are allowed.

There are so many terms in software which are nonsensical (starting with "computer science") which could be fixed. The problem with changing whitelist to "allowlist" is that it implies that people who use whitelist are racists. You're not just virtue signaling (and confusing my spellchecker) but causing discord. It would be perfectly fine if people switched to "allowlist" because they think it's a better term, but th…

I think that you are right. "Allow list" (or, in some contexts, "inclusion list") would be a better term, especially in contexts where colours are involved and it would be confusing.

The reasons that they usually actually have are not very good though, like you say, but nevertheless sometimes it can result in something better and sometimes not. But, banning words is not the solution, though.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#227

Post by johncolanduoni »

ctoth wrote: post #217

The web doesn't need attestation. It doesn't need signed agents. It doesn't need Cloudflare deciding who's a "real" user agent. It needs people to remember that "public" means PUBLIC and implement basic damn rate limiting if they can't handle the traffic. The web doesn't need to know if you're a human, a bot, or a dog. It just needs to serve bytes to whoever asks, within reasonable resource constraints. That's it. Th…

Basic damn rate limiting is pretty damn exploitable. Even ignoring botnets (which is impossible), usefully rate limiting IPv6 is anything but basic. If you just pick some prefix from /48 to /64 to key your rate limits on, you'll either be exploitable by IPs from providers that hand out /48s like candy or you'll bucket a ton of mobile users together for a single rate limit.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#228

Post by lucb1e »

quectophoton wrote: post #106
gausswho wrote: post #71

Earlier quoted context omitted.

What we need is some legal teeth behind robots.txt. It won't stop everyone, but Big Corp would be a tasty target for lawsuits.

I don't know about this. This means I'd get sued for using a feed reader on Codeberg[1], or for mirroring repositories from there (e.g. with Forgejo), since both are automated actions that are not caused directly by a user interaction (i.e. bots, rather than user agents). [1]: https://codeberg.org/robots.txt#:~:text=Disallow:%20/.git/,....

You don't get sued for using a service as it is meant to be used (using an RSS reader on their feed endpoint; cloning repositories that it is their mission to host). It doesn't anger anyone so they wouldn't bother trying to enforce a rule, and secondly it's a fruitless case because the judge would say it's not a reasonable claim they're making

Robots.txt is meant for crawlers, not user agents such as a feed reader or git client

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#229

Post by wvenable »

gradstudent wrote: post #180
wilson090 wrote: post #176

Earlier quoted context omitted.

How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.

How is it available for everyone if the AI bots bring down your server?

Is that really the problem we are discussing? I've had people attack my server and bring it down. But that has nothing to do with being free and open to everyone. A top hacker news post could take my server.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#230

Post by wvenable »

pton_xd wrote: post #197
wilson090 wrote: post #176

Earlier quoted context omitted.

How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.

> If you don't want AI bots reading information on the web, you don't actually want a free and open web. This is such a bad faith argument. We want a town center for the whole community to enjoy! What, you don't like those people shooting up drugs over there? But they're enjoying it too, this is what you wanted right? They're not harming you by doing their drugs. Everyone is enjoying it!

If an AI bot is accessing my site the way that regular users are accessing my site -- in other words everyone is using the town center as intended -- what is the problem?

Seems to be a lot of conflating of badly coded (intentionally or not) scrapers and AI. That is a problem that predates AI's existence.

Post reply on HN