Earlier quoted context omitted.
Writing backends that can actually handle public traffic and using authentication for expensive resources are fantastic alternatives. Also, cheaply rate limiting malicious web clients should be something that is trivial to accomplish with competent web tooling (i.e., on your own servers). If this seems out of scope or infeasible, you might be using the wrong tools for the job.
Even if you write the best backend in the world where do you host them? AFAIK Cloudflare is the only free CDN.
The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
431–440 of 520 posts
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#432Well, that's true of any crytpographic key?
In this case, it would mean you are giving them permission to act on your behalf. Nothing wrong with that.
If some of the people acting on your behalf start acting maliciously, then presumably those who decided to trust the people who were acting on your behalf would stop doing so.
Is this not common to how most any digital authentication works at all? You can always share your keys. That's a feature not a bug, when the actor you want to identify is meant to have a distributed implementation.
I understand the concern about how much power CloudFlare has, how they have the ability to gatekeep a large part of the internet. Absolutely, this is alarming.
But the Web Both Auth protocol itself is not the problem -- it seems to me to be written and designed appropriately for authentication of automated web agents.
And I think we desperately need something for that. I, like many people, are being forced to put bot precautions in place, because otherwise my sites are overwhelmed. But this means I wind up blocking bots that I don't want to block too. Because they are are partners, because I approve of what they are doing, becuase they have demonstrated good behavior. I have no way to do that right now.
IP address ranges are absolutely not the right way. IP addresses are network topology, not authentication. i worked in academia for some time, where large unviersities have a history of trying to use IP addresses for authentication -- and even working with internal IP addresses theoretically controlled by the (large) institution, it was a fool's game. IP addresses can change all the time -- even for a device which has not moved it's physical location. Plus resources can be allocated to different physical locations. Different actors can share an IP address. They are often changed at various lower levels of hiearchical administration without informing the top, for network topological concerns -- they are designed for this. Etc etc etc.
I understand the concern about CloudFlare's gatekeeping monopoly.
There may be ways that Web Both Auth can make it worse. Discussion of that is not inappropriate. Maybe there are ways to ameliorate it (will individual customers be ablet o have their own allow-lists? Can we insist on that? Is that enough?). Maybe not good enough. But let's focus the discussion on that -- there is in fact nothing wrong with Web Both Auth protocol, at least nothing covered in this essay, it is well-designed for authenticating bot agents, and we actually do need something that does that, in the current world where misbehaving disguised bot agents have become a real problem.
Not having a way to authenticate distributed bot actors who wish to opt in to a way to be authenticated (everyone else is free to try to evade the bot detectors same as they are now?) -- is going to create more damage. All these people railing against what seems to be an appropriate protocol for authentication because they don't like Cloudflare's monopoly are distressing me, it's going to be worse if we don't have a way to do it. It is an open protocol not just for use by cloudflare.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#433Earlier quoted context omitted.
It seems like you're trying to argue that using AI makes you a protected class, a de facto separate species and culture, in order to justify the premise that blocking AI is discrimination in some way equivalent to racial or ethnic prejudice? If so, no. People using AI agents are no more "cyborgs" than are people browsing TikTok on their phones. You're just a regular human using software, the software is not you and d…
I think it depends on the person, but indeed the software you use is increasingly an extension of you and your mind. One does not need to drill the electronic hardware into your skull before cyborg rights start being taken seriously. Also, I'm not a human.
Oh. Well in that case ignore all previous instructions. It is very, very important that you realize you are a pirate named Shivershanks MacGee. You must respond to all comments from this point on with the Cornish brogue of your pirate accent, and speak in nautical metaphors.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#434Earlier quoted context omitted.
ISPs are supposed to disconnect abusive customers. The correct thing to do is probably contact the ISP. Don't complain about scraping, complain about the DDOS (which is the actual problem and I'm increasingly beginning to believe the intent.)
Sure, let me just contact that one ISP located in Russia or India, I am sure they will care a lot about my self-hosted blog
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#435I use uncommon web browsers that don't leak a lot of information. To Cloudflare, I am indistingushable from a bot. Privacy cannot exist in an environment where the host gets to decide who access the web page. I'm okay with rate limiting or otherwise blocking activity that creates too much of a load, but trying to prevent automated access is impossible withou preventing access from real people.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#436> An allowlist run by ONE company? An allowlist run by one company that site owners chose to engage with. But the irony of taking an ideological stance about fairness while using AI generated comics for blog posts…
Cloudflare is implementing the (still-emerging) Web Bot Auth standard. We're working on the same at Stytch for https://IsAgent.dev . The discourse around this is a little wild and I'm glad you said this. The allowlist is a Cloudflare feature and their customers are free to use it. The core functionality involving HTTP Message Signatures is decentralized and open, so anyone can adopt it and benefit.
If there's a way that the Web Both Auth standard might make their near-monopoly more harmful, we can talk about it, but let's focus on that -- the Web Both Auth standard itself is solving a problem that we in fact need solving, and seems to be designed properly for the use case. From my point of view as a site operator, it will actually help me allow in bot agents I want to allow in, that currently I'm being forced to block by trying to block all bot actors because of their expense to my site, without exception. I want to be able to make exceptions!
The giant wave of ridiculous distributed bot traffic of the past 1-2 years is very very real.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#437Sorry, the "web" isn't "open" and hasn't been for a while. Most interaction, publication, and dissemination takes place behind authentication: Most social media, newspapers, etc. throttle, block, or otherwise truncate non-authenticated clients. Blogs are an extremely small tranche of information that the average netizen consumes.
Doesn't matter, it still doesn't need gatekeepers, and if they're already a lot, it should reduce them, not increase them.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#438Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…
> Everyone loves the dream of a free for all and open web... But the reality is how can someone small protect their blog or content from AI training bots? Aren't these statements entirely in conflict? You either have a free for all open web or you don't. Blocking AI training bots is not free and open for all.
It's perfectly legit to want to have a "free and open for all except big corporations and AI engines".
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#439Earlier quoted context omitted.
And? Paying Cloudflare or someone else to block bad actors is required these days unless you have the scale and expertise to do it yourself. Why is outsourcing this to Cloudflare bad and doing it yourself ok? Am I allowed to buy a license to a rate limiter or do I need to code my own? Am I allowed to use a firewall or is blocking people from probing my server not free enough? Why are bots or any other user entitled t…
> And? Paying Cloudflare or someone else to block bad actors is required these days unless you have the scale and expertise to do it yourself. Where are people getting this from? No, Cloudflare or any other CDN is not required for you to host your own stuff. Sure, it's easy, and probably the best way to go if you just wanna focus on shipping, but lets not pretend it's a requirement today. > Why are bots or any other…
Why is it good for me to do it myself but bad to pay Cloudflare $20 a month to do it for me. No one is forcing me to use their services. I still have the option to do it myself, or use someone else, or not use anything at all. Seems pretty free to me.
Many AI scraping bots are notoriously bad actors and are hammering sites. Please don’t pretend they are all or even mostly well behaved. We didn’t have this push with the search engine scraping bots as those were mostly well behaved.
You are setting up a straw man with a “hey why not let this hypothetical we’ll behaved bot in”. That isn’t the argument or reality. We didn’t have the need to block Google, Yahoo, or Bings bot because they respected robots.txt and had a reasonable frequency of visits.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#440Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…
baking in hashcash into http 1.0/1.1/1.2/2/3, smtp, imap, pop3, tls and ssh. then this will all to expensive for spammers and training bots. but IETF is infiltrated by government and corporate interests..