Live data from Hacker News

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

positiveblue.substack.com

391–400 of 520 posts

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#391
post #12

I pretty much use Perplexity exclusively at this point, instead of Google. I'd rather just get my questions answered than navigate all of the ads and slowness that Google provides. I'm fine with paying a small monthly fee, but I don't want Cloudflare being the gatekeeper. Perhaps a way to serve ads through the agents would be good enough. I'd prefer that to be some open protocol than controlled by a company.

Perplexity is the problem Cloudflare and companies like it are trying to solve. The company refuses to take no for an answer and will mislead and fake their way through until they've crawled the content they wanted to crawl.

The problem isn't just that ads can't be served. It's that every technical measure to attempt to block their service produces new ways of misleading website owners and the services they use. Perplexity refuses any attempt at abuse detection and prevention from their servers.

None of this would've been necessary if companies like Perplexity would've just acted like a responsible web service and told their customers "sorry, this website doesn't allow Perplexity to act on your behalf".

The open protocol you want already exists: it's the user agent. A responsible bot will set the correct user agent, maybe follow the instructions in robots.txt, and leave it at that. Companies like Perplexity (and many (AI) scrapers) don't want to participate in such a protocol. They will seek out and abuse any loopholes in any well-intended protocol anyone can come up with.

I don't think anyone wants Cloudflare to have even more influence on the internet, but it's thanks to the growth of inconsiderate AI companies like Perplexity that these measure are necessary. The protocol Cloudflare proposes is open (it's just a signature), the problem people have with it is that they have to ask Cloudflare nicely to permit website owners to track and prevent abuse from bots. For any Azure-gated websites, your bot would need to ask permission there as well, as with Akamai-gated websites, and maybe even individual websites.

A new protocol is a technical solution. Technical solutions work for technical problems. The problem Cloudflare is trying to solve isn't a technical problem; it's a social problem.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#392
post #327

Earlier quoted context omitted.

Shouldn't this be seen as success? You weren't a normal user, you were trying to penetrate the site, and you got a bunch of friction?

On the first fricking pageload I got blocked and couldn't open it at all, no captcha shown. That's a success only insofar as you want to exclude random people who don't have a second person whose cookie state to copy Also mind that not every request we make is malicious. A lot of it is also seeing what's even there, doing baseline requests, normal things. I didn't get the impression that I got blocked more on malicio…

Some websites will detect a Burp proxy and act accordingly. If you did your initial page load with any kind of integration like that, that's why the WAF may have blocked your request. I don't know exactly how they do it (my guess is fingerprinting the TLS handshake and TCP packet patterns), but I have seen several services do a great job at blocking any kind of analyzing proxy.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#393
post #7

I understand the concerns around a central gatekeeper but I'm confused as to why this specifically is viewed negatively. Don't website owners have to choose to enable cloudflare and to opt-in to this gate that the site owners control? If this was cloudflare going into some centralized routing of the internet and saying everything must do X then that would be a lot more alarming but at the end of the day the internet…

Site owners are tricked and scared (by Cloudflare) into using Cloudflare when they don't need to. Cloudflare feels the increase in customer growth and the rest of us feel the pain.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#394

Are bots using a large number of IP addresses simultaneously, so they look like a DDOS attack? Or are they just making ordinary requests from a small number of addresses. If it's the latter, all you need is some kind of fair queuing so those requests compete with each other for access, not with other users.

Bots are probing for access from various servers, eventually falling back to executing requests from residential IP addresses: https://blog.cloudflare.com/perplexity-is-using-stealth-unde...

Cloudflare is dealing with a couple million faked requests every day just from Perplexity users, and Perplexity is far from the worst player in the field.

The problem would be quite easy to solve with basic rate limiting if it weren't for the attempts to bypass access controls.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#395

I think it shouldn't require registering /with/ cloudflare. cloudflare should just look up the .well-known referenced and double check for impersonation, and keep score on how well behaved each one is.

Using completely automated means would leave open the possibility to set up a new signature for every single request, or for batches of requests. The manual step is to cut down on the amount of automated abuse.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#396

Earlier quoted context omitted.

At present, problem one is almost entirely AI companies.

There's actually not much evidence of this, since the attack traffic is anonymous.

HN people working in these AI companies have commented to say they do this, and the timing correlates with the rise of AI companies/funding.

I haven't tried to find it in my own logs, but others have said blocking an identifiable AI bot soon led to the same pattern of requests continuing through a botnet.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#397

Earlier quoted context omitted.

Google (and the others) crawl from a published IP range, with "Google" in the user agent. They read robots.txt. They are very easy to block The AI scum companies crawl from infected botnet IPs, with the user agent the same as the latest Chrome or Safari.

Okay. Which, specifically, are the "AI scum" companies you're speaking of? There are plenty of non-AI companies that also use dubiously sourced IPs and hide behind fake User-Agents.

I don't know which companies, of course. They hide their identity by using a botnet.

This traffic is new, and started around when many AI startups started.

I see traffic from new search engines and other crawlers, but it generally respects robots.txt and identifies itself, or else comes from a small pool of IP addresses.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#398
post #21

Earlier quoted context omitted.

Writing backends that can actually handle public traffic and using authentication for expensive resources are fantastic alternatives. Also, cheaply rate limiting malicious web clients should be something that is trivial to accomplish with competent web tooling (i.e., on your own servers). If this seems out of scope or infeasible, you might be using the wrong tools for the job.

If it were this easy, we wouldn't have had about 10 HN posts on the topic in the last few months.

The technical skills of the majority of the HN community are way below those of the typical computing community a generation ago.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#399
post #9
post #4

Earlier quoted context omitted.

AWS is an alternative no?

AWS needs a dedicated AWS engineer while any technical person and some non-technical people have skill to set up Cloudflare. Esp. Without surprise bills.

I thought the whole point of paying a fortune for AWS was to avoid having a dedicated engineer. It’s the cobol of the 21st century.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#400

We have far too many gatekeepers as it is. Any attempt to add any more should be treated as an act of aggression. Cloudflare seems very vocal about its desire to become yet another digital gatekeeper as of late, and so is Google. I want both reduced to rubble if they persist in it.

What do you mean Google "desires" to become a gatekeeper? They have been a gatekeeper for years, since they control the browser everyone uses, and Firefox usage is now in the noise. Google just steers the www where they want it to go. Killing ublock, pushing .webp trash, etc.
Post reply on HN