Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

positiveblue.substack.com

Post reply on HN
Print view View original

381–390 of 520 posts

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#381

Post by Voultapher »

impure wrote: post #16

Well, if you have a better way to solve this that’s open I’m all ears. But what Cloudflare is doing is solving the real problem of AI bots. We’ve tried to solve this problem with IP blocking and user agents, but they do not work. And this is actually how other similar problems have been solved. Certificate authorities aren’t open and yet they work just fine. Attestation providers are also not open and they work just…

> Well, if you have a better way to solve this that’s open I’m all ears.

Regulation.

Make it illegal to request the content of a webpage by crawler if a website operator doesn't explicitly allows it via robots.txt. Institute a government agency that is tasked with enforcement. If you as a website operator can show that traffic came from bots, you can open a complaint with the government agency and they take care of shaking painful fines out of the offending companies. Force cloud hosts to keep books on who was using what IP addresses. Will it be a 100% fix, no, will it have a massive chilling effect if done well, absolutely.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#382

Post by jeroenhd »

ACCount37 wrote: post #29

We have far too many gatekeepers as it is. Any attempt to add any more should be treated as an act of aggression. Cloudflare seems very vocal about its desire to become yet another digital gatekeeper as of late, and so is Google. I want both reduced to rubble if they persist in it.

Several companies are looking to provide a solution for the AI bot problem. Cloudflare stands to make a lot of money if people pick their solution. But Cloudflare backing down won't make the problem go away, and someone else's bad solution will be chosen instead.

The gatekeeping described here is gatekeeping a website owner chooses. It's an alternative to pay walls, bespoke bot detection, or some kind of ID verification. Cloudflare already provides a service, but standardising the service will open up the market (at the cost of competitors adopting Cloudflare's standard).

The freedom of the open web also extends to the owners of the websites people visit.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#383

Post by jeroenhd »

sdsd wrote: post #6

Maybe the title means something more like "The web should not have gatekeepers (Cloudflare)". They do seem to say as much toward the end: >We need protocols, not gatekeepers. But until we have working protocols, many webmasters literally do need a gatekeeper if they want to realistically keep their site safe and online. I wish this weren't the case, but I believe the "protocol" era of the web was basically ended when…

>We need protocols, not gatekeepers

The funny thing is that this blog post is complaining about a proposed protocol from Cloudflare (one which will identify bots so that good bots can be permitted). The signup form is just a method to ask Cloudflare (or any other website owner/CDN) to be categorized as a good bot.

It's not a great protocol if you're in the business of scraping websites or selling people bots to access websites for them, but it's a great protocol for people who just want their website to work without being overwhelmed by the bad side of the internet.

The whitelist approach Cloudflare takes isn't good for the internet, but for website owners who are already behind Cloudflare, it's better than the alternative. Someone will need to come up with a better protocol that also serves the website owners' needs if they want Cloudflare to fail here. The AI industry simply doesn't want to cooperate, so their hand must be forced, and only companies like Cloudflare are powerful enough to accomplish that.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#384

Post by jeroenhd »

neximo64 wrote: post #341

So Cloudflare becomes the gatekeeper then? I kind of want my site to be indexed with agents and used without any interference

By not using Cloudflare your website will be indexed by everyone. The gatekeeper aspect only applies if you use Cloudflare to distribute your website (and even then Cloudflare offers options to control this bot shield thing).

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#385

Post by thrown-0825 »

Voultapher wrote: post #381
impure wrote: post #16

Well, if you have a better way to solve this that’s open I’m all ears. But what Cloudflare is doing is solving the real problem of AI bots. We’ve tried to solve this problem with IP blocking and user agents, but they do not work. And this is actually how other similar problems have been solved. Certificate authorities aren’t open and yet they work just fine. Attestation providers are also not open and they work just…

> Well, if you have a better way to solve this that’s open I’m all ears. Regulation. Make it illegal to request the content of a webpage by crawler if a website operator doesn't explicitly allows it via robots.txt. Institute a government agency that is tasked with enforcement. If you as a website operator can show that traffic came from bots, you can open a complaint with the government agency and they take care of s…

[flagged]

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#386

Post by jeroenhd »

zzo38computer wrote: post #240

With what they say about authorization, I think X.509 would help. (Although central certificate authorities are often used with X.509, it does not have to be that way; the service you are operating can issue the certificate to you instead, or they can accept a self-signed certificate which is associated with you the first time it is used to create an account on their service.) You can use the admin certificate issued…

What problem does this solve that a basic API key doesn't solve already? The issue with that approach is that you will require accounts/keys/certificates for all hosts you intend to visit, and malicious bots can create as many accounts as they need. You're just adding a registration step to the crawling process.

Your suggested approach works for websites that want to offer AI access as a service to their customers, but the problem Cloudflare is trying to solve is that most AI bots are doing things that website owners don't want them to do. The goal is to identify and block bad actors, not to make things easier for good actors.

Using mTLS/client certificates also exposes people (that don't use AI bots) to the awful UI that browsers have for this kind of authentication. We'll need to get that sorted before an X509-based solution makes any sense.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#387

Post by neximo64 »

jeroenhd wrote: post #384
neximo64 wrote: post #341

So Cloudflare becomes the gatekeeper then? I kind of want my site to be indexed with agents and used without any interference

By not using Cloudflare your website will be indexed by everyone. The gatekeeper aspect only applies if you use Cloudflare to distribute your website (and even then Cloudflare offers options to control this bot shield thing).

I want it to be indexed by everyone, thats the whole point.

So what then Cloudflare can use all these websites as leverage against Google, OpenAI and Microsoft? I kind of want my content to be indexed.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#388

Post by jeroenhd »

chrisweekly wrote: post #140
seanvelasco wrote: post #43

as a Cloudflare customer, I am happy with their proposition. I personally do not want companies like Perplexity that fake their user-agent and ignore my robots.txt to trespass. and isn't this why people sign up with Cloudflare in the first place? for bot protection? to me, this is just the same, but with agents. i love the idea of an open internet, but this requires all party to be honest. a company like Perplexity t…

Your complaints about "faking their user-agent" reminds me of this 15-year-old but still-relevant, classic post about the history of the user-agent string: https://webaim.org/blog/user-agent-string-history/ TLDR the UA string has always been "faked", even in the scenarios you might think are most legitimate.

The traditional UA fakery (adding Mozilla to the start and then just tacking on browser engine names) was the result of outdated websites breaking browsers.

The problematic fakery here is that bots are pretending to be people by emulating browsers to prevent rate limits and other technical controls.

That second category has also been with us since the dawn of the internet, but it has always been something worth complaining about. No trustworthy tool or service will pretend to be a real browser, at least not by default.

If AI agents just identified themselves as such, we wouldn't need elaborate schemes to block them when they need to be blocked.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#389

Post by jeroenhd »

neximo64 wrote: post #387
jeroenhd wrote: post #384

Earlier quoted context omitted.

By not using Cloudflare your website will be indexed by everyone. The gatekeeper aspect only applies if you use Cloudflare to distribute your website (and even then Cloudflare offers options to control this bot shield thing).

I want it to be indexed by everyone, thats the whole point. So what then Cloudflare can use all these websites as leverage against Google, OpenAI and Microsoft? I kind of want my content to be indexed.

The content you host will only be blocked from being indexed if you decide to use a service that blocks indexing. If you host your content on other people's services, then you never had the power to make that decision anyway.

If you want your content to be indexed, simply don't use Cloudflare. Host your own servers. Use a different CDN if you want the benefits of Cloudflare's networks.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#390

Post by ccgreg »

jeroenhd wrote: post #383
sdsd wrote: post #6

Maybe the title means something more like "The web should not have gatekeepers (Cloudflare)". They do seem to say as much toward the end: >We need protocols, not gatekeepers. But until we have working protocols, many webmasters literally do need a gatekeeper if they want to realistically keep their site safe and online. I wish this weren't the case, but I believe the "protocol" era of the web was basically ended when…

>We need protocols, not gatekeepers The funny thing is that this blog post is complaining about a proposed protocol from Cloudflare (one which will identify bots so that good bots can be permitted). The signup form is just a method to ask Cloudflare (or any other website owner/CDN) to be categorized as a good bot. It's not a great protocol if you're in the business of scraping websites or selling people bots to acces…

Conventional crawlers already have a way to identify themselves, via a json file containing a list of IP addresses. Cloudflare is fully aware of this defacto standard.
Post reply on HN