Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

positiveblue.substack.com

Post reply on HN
Print view View original

131–140 of 520 posts

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#131

Post by quectophoton »

blibble wrote: post #121
quectophoton wrote: post #106

Earlier quoted context omitted.

I don't know about this. This means I'd get sued for using a feed reader on Codeberg[1], or for mirroring repositories from there (e.g. with Forgejo), since both are automated actions that are not caused directly by a user interaction (i.e. bots, rather than user agents). [1]: https://codeberg.org/robots.txt#:~:text=Disallow:%20/.git/,....

> This means I'd get sued for using a feed reader on Codeberg you think codeberg would sue you?

Probably not.

But it's the same thing with random software from a random nobody that has no license, or has a license that's not open-source: If I use those libraries or programs, do I think they would sue me? Probably not.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#132

Post by supriyo-biswas »

Klonoar wrote: post #125
supriyo-biswas wrote: post #48

Earlier quoted context omitted.

Well there's open source stuff like https://github.com/TecharoHQ/anubis ; one doesn't need a top-down mandated solution coming from a corporation. In general Cloudflare has been pushing DRMization of the web for quite some time, and while I understand why they want to do it, I wish they didn't always show off as taking the moral high ground.

Anubis doesn’t necessarily stop the most well funded actors. If anything we’ve seen the rise in complaints about it just annoying average users.

The actual response to which Anubis was created is seemingly a strange kind of DDOS attack that has been misattributed to LLMs, but is some kind of attacker that makes partial GET requests that are aborted soon after sending the request headers, mostly coming from residential proxies. (Yes, it doesn’t help that the author of Anubis also isn’t fully aware of the mechanics of the attack. In fact, there is no proper write up of the mechanism of the attack which I hope to write about someday).

Having said that, the solution is effective enough, having a lightweight proxy component that issues proof of work tokens to such bogus requests works well enough, as various users on HN seem to point out.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#133

Post by swed420 »

matt-p wrote: post #5

I have zero issue with Ai Agents, if there's a real user behind there somewhere. I DO have a major issue with my sites being crawled extremely aggressively by offenders including Meta, Perplexity and OpenAI - it's really annoying realising that we're tying up several cpu cores on AI crawling. Less than on real users and google et al.

> I DO have a major issue with my sites being crawled extremely aggressively by offenders including Meta, Perplexity and OpenAI

Gee, if only we had, like, one central archive of the internet. We could even call it the internet archive.

Then, all these AI companies could interface directly with that single entity on terms that are agreeable.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#134

Post by Gud »

victorbjorklund wrote: post #120
Gud wrote: post #27

Earlier quoted context omitted.

By developing Free Software combating these hostile softwares. Corporations develop hostile AI agents, Capable hackers develop anti-AI-agents. This defeatist atittude "we have no power".

So basically cloudflare but self-hosted (with all the pain that comes from that)?

What’s so painful about self hosting? I’ve been self hosting since before I hit puberty. If 12 year old me can run a httpd, anyone can.

And if you don’t want to self host, at least try to use services from organisations that aren’t hostile to the open web

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#135

Post by ehnto »

dlcarrier wrote: post #60

I use uncommon web browsers that don't leak a lot of information. To Cloudflare, I am indistingushable from a bot. Privacy cannot exist in an environment where the host gets to decide who access the web page. I'm okay with rate limiting or otherwise blocking activity that creates too much of a load, but trying to prevent automated access is impossible withou preventing access from real people.

I also do the same and get caught up by bot blockers.

However, I do believe the host can do whatever they want with my request also.

This issue becomes more complex when you start talking about government sites, since ideally they have a much stronger mandate to serve everyone fairly.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#136

Post by verdverm »

SoftTalker wrote: post #102
verdverm wrote: post #72

Earlier quoted context omitted.

The website owner has rights too. Are you arguing they cannot choose to implement such gatekeeping to keep their site operating in a financially viable manner?

If you put your information freely on the web, you should have minimal expectations on who uses it and how. If you want to make money from it, put up a paywall. If you want the best of both worlds, i.e. just post freely but make money from ads, or inserting hidden pixels to update some profile about me, well good luck. I'll choose whether I want to look at ads, or load tracking pixels, and my answer is no.

I'm not talking about ads or pixels, I'm referring to bot operators creating so much traffic that the network bill makes the hosting financially impossible

> my answer is no.

Rights for me, but not for thee?

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#137

Post by victorbjorklund »

Gud wrote: post #134
victorbjorklund wrote: post #120

Earlier quoted context omitted.

So basically cloudflare but self-hosted (with all the pain that comes from that)?

What’s so painful about self hosting? I’ve been self hosting since before I hit puberty. If 12 year old me can run a httpd, anyone can. And if you don’t want to self host, at least try to use services from organisations that aren’t hostile to the open web

I self-host lots of stuff. But yes it is more pain to host a WAF that can handle billions of request per minute. Even harder to do it for free like Cloudflare. And in the end the end result for the user is exactly the same if you use a self-hosted WAF or let someone else host it for you.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#138

Post by [deleted] »

Gud wrote: post #134
victorbjorklund wrote: post #120

Earlier quoted context omitted.

So basically cloudflare but self-hosted (with all the pain that comes from that)?

What’s so painful about self hosting? I’ve been self hosting since before I hit puberty. If 12 year old me can run a httpd, anyone can. And if you don’t want to self host, at least try to use services from organisations that aren’t hostile to the open web

[deleted]

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#139

Post by avazhi »

TIPSIO wrote: post #20

Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…

Nobody cares about robots.txt, nor should they.

If this is your primary argument against being scraped (viz that your robots.txt said not to) then you’re naive and you’re doing it wrong.

If the internet is open, then data on it is going to be scraped lol. You can’t have it both ways.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#140

Post by chrisweekly »

seanvelasco wrote: post #43

as a Cloudflare customer, I am happy with their proposition. I personally do not want companies like Perplexity that fake their user-agent and ignore my robots.txt to trespass. and isn't this why people sign up with Cloudflare in the first place? for bot protection? to me, this is just the same, but with agents. i love the idea of an open internet, but this requires all party to be honest. a company like Perplexity t…

Your complaints about "faking their user-agent" reminds me of this 15-year-old but still-relevant, classic post about the history of the user-agent string:

https://webaim.org/blog/user-agent-string-history/

TLDR the UA string has always been "faked", even in the scenarios you might think are most legitimate.

Post reply on HN