The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch • Hacker News Board

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#511

Post by account42 » Mon, Sep 01, 2025, 10:52 AM UTC

palmfacehn wrote: post #35

matt-p wrote: post #28

Earlier quoted context omitted.

Honestly it's just tragedy of the commons. Why put the effort in when you don't have to identify yourself, just crawl and if you get blocked move the job to another server.

At this point I'm blocking several ASNs. Most are cloud provider related, but there are also some repurposed consumer ASNs coming out of the PRC. Long term, this devalues the offerings of those cloud providers, as prospective customers will not be able to use them for crawling.

This is the correct solution and is how network abuse has been dealt with before the latest fad. Network operators can either police their own users or be blocked/throttled wholesale. There isn't anything more needed except for the willingness to apply measures to networks that are "too big to fail".

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#512

Post by account42 » Mon, Sep 01, 2025, 11:03 AM UTC

jlarocco wrote: post #457

Voultapher wrote: post #381

Earlier quoted context omitted.

> Well, if you have a better way to solve this that’s open I’m all ears. Regulation. Make it illegal to request the content of a webpage by crawler if a website operator doesn't explicitly allows it via robots.txt. Institute a government agency that is tasked with enforcement. If you as a website operator can show that traffic came from bots, you can open a complaint with the government agency and they take care of s…

I'm not anti-government, but a technical solution that elliminates the the problem is infinitely better than regulating around it. The internet is too big and distributed to regulate. Nobody will agree on what the rules should be, and certain groups or countries will disagree in any case and refuse to enforce them. Existing regulation rarely works, and enforcement is half-assed, at best. Ransomware is regulated and i…

The problem is that a technical solution is impossible.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#513

Post by account42 » Mon, Sep 01, 2025, 11:07 AM UTC

zimmund wrote: post #473

Voultapher wrote: post #381

Earlier quoted context omitted.

> Well, if you have a better way to solve this that’s open I’m all ears. Regulation. Make it illegal to request the content of a webpage by crawler if a website operator doesn't explicitly allows it via robots.txt. Institute a government agency that is tasked with enforcement. If you as a website operator can show that traffic came from bots, you can open a complaint with the government agency and they take care of s…

> Institute a government agency that is tasked with enforcement. You're forgetting about the first W in WWW...

So what you're saying is that if I were to host a bit torrent tracker in Sweden then the US can't do anything about it?

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#514

Post by account42 » Mon, Sep 01, 2025, 11:12 AM UTC

esseph wrote: post #77

viktorcode wrote: post #68

Earlier quoted context omitted.

AI poisoning is a better protection. Cloudflare is capable of serving stashes of bad data to AI bots as protective barrier to their clients.

AI poisoning is going to get a lot of people killed, be cause the AI won't stop being used.

This is some next level blame shifting. Next you are going to steal motor oil and then complain that your customers got sick when you used it to cook their food.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#515

Post by account42 » Mon, Sep 01, 2025, 11:14 AM UTC

jeroenhd wrote: post #382

ACCount37 wrote: post #29

We have far too many gatekeepers as it is. Any attempt to add any more should be treated as an act of aggression. Cloudflare seems very vocal about its desire to become yet another digital gatekeeper as of late, and so is Google. I want both reduced to rubble if they persist in it.

Several companies are looking to provide a solution for the AI bot problem. Cloudflare stands to make a lot of money if people pick their solution. But Cloudflare backing down won't make the problem go away, and someone else's bad solution will be chosen instead. The gatekeeping described here is gatekeeping a website owner chooses. It's an alternative to pay walls, bespoke bot detection, or some kind of ID verificat…

You could use the same argument against antitrust laws. All monopolies were "chosen" by their customers, that doesn't mean we should give them a free pass.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#516

Post by ManlyBread » Mon, Sep 01, 2025, 5:12 PM UTC

lotsofpulp wrote: post #434

ManlyBread wrote: post #428

Earlier quoted context omitted.

Sure, let me just contact that one ISP located in Russia or India, I am sure they will care a lot about my self-hosted blog

Hence the need for Cloudflare?

I am not comfortable with a private company being the only solution, especially when they have a history of deplatforming sites.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#517

Post by ManlyBread » Mon, Sep 01, 2025, 5:12 PM UTC

account42 wrote: post #499

ManlyBread wrote: post #428

Earlier quoted context omitted.

Sure, let me just contact that one ISP located in Russia or India, I am sure they will care a lot about my self-hosted blog

Except that's exactly what you should do. And if they refuse to cooperate you contact the network operators between them and yourself. Imagine if Chinese or Russian criminal gangs started sending mail bombs to the US/EU and our solution would be to require all senders, including domestic ones, to prove their identity in order to have their parcels delivered. Completely absurd, but somehow with the Internet everyone j…

The internet is not a mirror of the real world.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#518

Post by lucb1e » Mon, Sep 01, 2025, 8:43 PM UTC

PaulHoule wrote: post #445

lucb1e wrote: post #325

Earlier quoted context omitted.

They could. Let me know when it happens I have this argument every time self hosting comes up, and every time I wonder if someone will do it to me to make a point. Or if one of the like million other comments I post upsets someone or one of the many tools that I host. Yet to happen, idk. It's like arguing whether you need a knife on the street at all times because someone might get angry from a look. It happens, we h…

I've had my involvement with the computer underground. A web site owner published something he really shouldn't have and got hacked. I wound up being a "person of interest" in the resulting FBI investigation because I was the weirdest person in the chat room for the site. I think it drove them crazy I was using Tor so they got somebody to try to entrap me into sharing CP but (1) I'm not interested and (2) know better…

That's definitely the most interesting response I've had to this question, thanks for that

Will have to give this a second thought but as a first one now that I read this: ...and would Cloudflare have helped against the FBI, or any foreign nation doing a request with Cloudflare against child porn? Surely not?! A different kind of opsec is surely more relevant there, so I don't know if it's really relevant to "normal", legal self hosting (as opposed to criminal, much less that level of unethical+criminal) communities or if there's an aspect I'm missing here

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#519

Post by raxxorraxor » Tue, Sep 02, 2025, 12:00 PM UTC

TIPSIO wrote: post #20

Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…

This isn't even an issue, you have made that problem up. I host a blog and there are some AI bots coming around. Big deal. Most of them do respect a robots.txt. Some don't. Not a big deal as well.

In contrast trying to change the infrastructure of the net, which previously was quite resistant to censorship is quite a big deal.

This sounds exactly like a crazy preacher warning about the dangers of rock music. A completely made up threat. And we need the protection of god against these evil AI bots.

Wow, a bot that disrespected a robots.txt. How can the internet survive...

Also, OpenAI already has the data. You want to ensure they will never get competitors by putting up barriers now. It makes no sense...

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#520

Post by calmbonsai » Fri, Sep 05, 2025, 1:12 AM UTC

kentonv wrote: post #495

calmbonsai wrote: post #494

Earlier quoted context omitted.

More precisely, Cloudflare should not be able to offer this service. AI agents should be blocked at the hosted API endpoint ("last mile"), not at a CDN or other sort of intermediary. That said, if you're using Cloudflare Workers, where the endpoint itself is hosted by Cloudflare, that would be ethical.

If the CDN is implementing the explicitly-configured preferences of the specific site, what difference does it make if the blocking happens at the CDN vs. the site's origin server?

If it's via a CDN pull, that's totally cool. IF it's with a push or acting as a pure MIM, that's not cool.