Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

positiveblue.substack.com

Post reply on HN
Print view View original

421–430 of 520 posts

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#421

Post by lostmsu »

Symbiote wrote: post #306
davepeck wrote: post #219

Earlier quoted context omitted.

> Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? I'm old enough to remember when people asked the same questions of Hotbot, Lycos, Altavista, Ask Jeeves, and -- eventually -- Google. Then, as now, it never felt like the right way to frame the question. If you want your content freely available, make it freely avail…

Google (and the others) crawl from a published IP range, with "Google" in the user agent. They read robots.txt. They are very easy to block The AI scum companies crawl from infected botnet IPs, with the user agent the same as the latest Chrome or Safari.

Why do you think the bots you see are AI scum companies?

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#422

Post by gck1 »

dlcarrier wrote: post #60

I use uncommon web browsers that don't leak a lot of information. To Cloudflare, I am indistingushable from a bot. Privacy cannot exist in an environment where the host gets to decide who access the web page. I'm okay with rate limiting or otherwise blocking activity that creates too much of a load, but trying to prevent automated access is impossible withou preventing access from real people.

And god forbid you live in an authoritarian country and must use VPN to protect your freedom. Internet becomes captcha hell run by 2-3 companies.

I've had far fewer issues with my own bots that access cloudflare protected websites, than during my regular browsing with privacy respecting browsers and a VPN.

As a side note: I'm at least thankful Microsoft isn't behind web gatekeeping. Try and solve any microsoft captcha behind a VPN - its like writing a thesis, you gotta dedicate like 5 minutes, full attention.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#423

Post by diggan »

selfmodruntime wrote: post #409
wvenable wrote: post #172

Earlier quoted context omitted.

> Everyone loves the dream of a free for all and open web... But the reality is how can someone small protect their blog or content from AI training bots? Aren't these statements entirely in conflict? You either have a free for all open web or you don't. Blocking AI training bots is not free and open for all.

Nothing is „free“. AI bots eat up my blog like crazy and I have to pay for its hosting.

Don't you have rate-limits? And how much are you paying for the instance where you're hosting it? I've run/helped run projects with something like ~10 req/s easily on $10 VPSs, surely hosting HTML can't cost you that much?

Of course it won't be free, but you can get pretty close to free but employing the typical things you'd put in place to restrict the amount of resources used, like rate-limits, caches and so on.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#424

Post by sumeno »

tiahura wrote: post #177
TIPSIO wrote: post #20

Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…

Why should your blog be protected? Information wants to be free.

It's amazing how this catchphrase has reversed meanings for some people. It was previously used against walled gardens and paywalls, but these corporate LLMs are the ultimate walled garden for information because in most cases you can't even find out who created the information in the first place.

"Information wants to be free! That's why I support hiding it behind a chatbot paywall that makes a few people billionaires"

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#425

Post by lostmsu »

ehnto wrote: post #135
dlcarrier wrote: post #60

I use uncommon web browsers that don't leak a lot of information. To Cloudflare, I am indistingushable from a bot. Privacy cannot exist in an environment where the host gets to decide who access the web page. I'm okay with rate limiting or otherwise blocking activity that creates too much of a load, but trying to prevent automated access is impossible withou preventing access from real people.

I also do the same and get caught up by bot blockers. However, I do believe the host can do whatever they want with my request also. This issue becomes more complex when you start talking about government sites, since ideally they have a much stronger mandate to serve everyone fairly.

I agree with you, but the website owners just don't seem to understand that they are making their small problem into a big problem for real people, some of which will drop off.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#426

Post by diggan »

alemanek wrote: post #256
wvenable wrote: post #172

Earlier quoted context omitted.

> Everyone loves the dream of a free for all and open web... But the reality is how can someone small protect their blog or content from AI training bots? Aren't these statements entirely in conflict? You either have a free for all open web or you don't. Blocking AI training bots is not free and open for all.

And? Paying Cloudflare or someone else to block bad actors is required these days unless you have the scale and expertise to do it yourself. Why is outsourcing this to Cloudflare bad and doing it yourself ok? Am I allowed to buy a license to a rate limiter or do I need to code my own? Am I allowed to use a firewall or is blocking people from probing my server not free enough? Why are bots or any other user entitled t…

> And? Paying Cloudflare or someone else to block bad actors is required these days unless you have the scale and expertise to do it yourself.

Where are people getting this from? No, Cloudflare or any other CDN is not required for you to host your own stuff. Sure, it's easy, and probably the best way to go if you just wanna focus on shipping, but lets not pretend it's a requirement today.

> Why are bots or any other user entitled to unlimited visits to my website? The entitlement is kind of unreal at this point

I don't think they are, that's why we have rate limiters, right? :) I think the point is that if you're allowing a user to access some content in one way, why not allow that same user to access the content in the same way, but using a different user-agent? That's the original purpose of that header after all, to signal what the user used as an agent on their behalf. Commonly, I use Firefox as my agent for browsing, but I should be free to use any user-agent, if we want the web to remain open and free.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#427

Post by gck1 »

nromiun wrote: post #105
skybrian wrote: post #18

This is sort of like how email is based on Internet standards but a large percentage of email users use Gmail. The Internet standards Cloudflare is promoting are open, but Cloudflare has a lot of power due to having so many customers. (What are some good alternatives to Cloudflare?) Another way the situation is similar: email delivery is often unreliable and hard to implement due to spam filters. A similar thing seem…

It is a big problem. There is no good alternative to Cloudflare as a free CDN. They put servers all over the world and they are giving them away for free. And making their money on premium serverless services. Not to mention the big cloud providers are unhinged with their egress pricing.

> Not to mention the big cloud providers are unhinged with their egress pricing.

I always wonder why this status quo persisted even after Cloudflare. Their pricing is indeed so unhinged, that they're not even in consideration for me for things where egress is a variable.

Why is egress seemingly free for Cloudflare or Hetzner but feels like they launch spaceships at AWS and GCP every time you send a data packet to the outside world?

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#428

Post by ManlyBread »

msgodel wrote: post #405
BrenBarn wrote: post #362

Earlier quoted context omitted.

The thing is that rhinoceroses aren't well-behaved. Even if some small fraction of them in theory might be well-behaved, the effort of trying to account for that is too small to bother. If 99% of rhinoceroses aren't well-behaved, the simple and correct response is to ban them all, and then maybe the nice ones can ask for a special permit. You switch from allow-by-default to block-by-default. Similarly it doesn't make…

ISPs are supposed to disconnect abusive customers. The correct thing to do is probably contact the ISP. Don't complain about scraping, complain about the DDOS (which is the actual problem and I'm increasingly beginning to believe the intent.)

Sure, let me just contact that one ISP located in Russia or India, I am sure they will care a lot about my self-hosted blog

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#429

Post by djoldman »

Sorry, the "web" isn't "open" and hasn't been for a while.

Most interaction, publication, and dissemination takes place behind authentication:

Most social media, newspapers, etc. throttle, block, or otherwise truncate non-authenticated clients.

Blogs are an extremely small tranche of information that the average netizen consumes.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#430

Post by nromiun »

gck1 wrote: post #427
nromiun wrote: post #105

Earlier quoted context omitted.

It is a big problem. There is no good alternative to Cloudflare as a free CDN. They put servers all over the world and they are giving them away for free. And making their money on premium serverless services. Not to mention the big cloud providers are unhinged with their egress pricing.

> Not to mention the big cloud providers are unhinged with their egress pricing. I always wonder why this status quo persisted even after Cloudflare. Their pricing is indeed so unhinged, that they're not even in consideration for me for things where egress is a variable. Why is egress seemingly free for Cloudflare or Hetzner but feels like they launch spaceships at AWS and GCP every time you send a data packet to the…

They are just greedy. And they know nobody can compete with them for availability in every country. Except for Cloudflare, which is why it is so popular.
Post reply on HN