Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

positiveblue.substack.com

Post reply on HN
Print view View original

471–480 of 520 posts

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#471

Post by h4ck_th3_pl4n3t »

TIPSIO wrote: post #20

Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…

The problem's cause really is about _who_ has to pay for the traffic, and currently that's the hosting end. If you turn that model around, suddenly AI web scrapers have to behave and all the issues that we currently have are kind of solved(?), because there is no incentive to scrape datasets anymore that were put together by others, and there automatically will be a payment incentive instead to buy high-quality datasets.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#472

Post by wvenable »

frm88 wrote: post #364
wvenable wrote: post #298

Earlier quoted context omitted.

But then we get to use those AI tools. The refrain here comes down not to "AI" but mostly to "the AI bot assault" which is a different thing. Sure lets have an discussion about badly behaved and overzealous web scrapers. As for credit, I've asked AI for it's references and gotten them. If my information is merely mushed into AI training model I'm not sure why I need credit. If you discuss this thread with your friend…

"If you discuss this thread with your friends are you going to give me credit?" Yes. How else would I enable my friends to look it up for themselves?

6 months from now when you've internalized this entire thread are you even going to remember where you got it from?

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#473

Post by zimmund »

Voultapher wrote: post #381
impure wrote: post #16

Well, if you have a better way to solve this that’s open I’m all ears. But what Cloudflare is doing is solving the real problem of AI bots. We’ve tried to solve this problem with IP blocking and user agents, but they do not work. And this is actually how other similar problems have been solved. Certificate authorities aren’t open and yet they work just fine. Attestation providers are also not open and they work just…

> Well, if you have a better way to solve this that’s open I’m all ears. Regulation. Make it illegal to request the content of a webpage by crawler if a website operator doesn't explicitly allows it via robots.txt. Institute a government agency that is tasked with enforcement. If you as a website operator can show that traffic came from bots, you can open a complaint with the government agency and they take care of s…

> Institute a government agency that is tasked with enforcement.

You're forgetting about the first W in WWW...

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#474

Post by swed420 »

teitoklien wrote: post #163
swed420 wrote: post #133

Earlier quoted context omitted.

> I DO have a major issue with my sites being crawled extremely aggressively by offenders including Meta, Perplexity and OpenAI Gee, if only we had, like, one central archive of the internet. We could even call it the internet archive. Then, all these AI companies could interface directly with that single entity on terms that are agreeable.

you think they care about that ? they’d still crawl like this just in case which is why they don’t rate limit atm

It would of course need to be legally enforced somehow, with penalties high enough to hurt even the big players.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#475

Post by swed420 »

gck1 wrote: post #420
swed420 wrote: post #133

Earlier quoted context omitted.

> I DO have a major issue with my sites being crawled extremely aggressively by offenders including Meta, Perplexity and OpenAI Gee, if only we had, like, one central archive of the internet. We could even call it the internet archive. Then, all these AI companies could interface directly with that single entity on terms that are agreeable.

Internet Archive is missing enormous chunks of the internet though. And I don't mean weird parts of the internet, just regional stuff. Not even news articles from top 10 news websites from my country are usually indexed there.

So then make a better one. I was only referencing it as a general concept that can be approved upon as desired.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#476

Post by esseph »

viktorcode wrote: post #161
esseph wrote: post #77

Earlier quoted context omitted.

AI poisoning is going to get a lot of people killed, be cause the AI won't stop being used.

By that logic AI already killing people. We can't presume that whatever can be found on the internet is reliable data, can't we?

Yet we use it every day for police, military, and political targeting with economic and kinetic consequences.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#477

Post by akoboldfrying »

BrenBarn wrote: post #463
akoboldfrying wrote: post #406

Earlier quoted context omitted.

> "If your grocery store is open to the public, why is it not open to this herd of rhinoceroses?" What this scenario actually reveals is that the words "open to the public" are not intended to mean "access is completely unrestricted". It's fine to not want to give completely unrestricted access to something. What's not fine, or at least what complicates things unnecessarily, is using words like "open and free" to des…

Using "open and free" to mean "I actually want no restrictions at all" is also confusing and disingenuous, because, as you yourself point out, a lot of people don't mean that by those words. The other thing, though, is that there's a difference between "I personally want to release my personal work under open, free, and unrestricted terms" and "I want to release my work into a system that allows people to access info…

> Asking people to just unilaterally disarm by imposing no restrictions

I'm not asking for this. I'm asking for people who want such restrictions (most of which I consider entirely reasonable) to say so explicitly. It would be enough to replace words like "free" or "open" with "fair use", which immediately signals that some restrictions are intended, without getting bogged down in details.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#478

Post by akoboldfrying »

Voultapher wrote: post #449
akoboldfrying wrote: post #414

Earlier quoted context omitted.

It just seems too easy to skirt around via middlemen. The EU (say) could prosecute an EU company directly doing this residential scraping, and it could probably keep tabs on a handful of bank accounts of known bad actors in other countries, and then investigate and prosecute EU companies transferring money to them. But how do you stop an EU company paying a Moldovan company (that has existed for 10 days) for "interne…

Genuinely this isn't a tech specific or even novel problem. There is plenty of prior art when it comes to inhibiting unwanted behavior. > But how do you stop an EU company paying a Moldovan company (that has existed for 10 days) for "internet services", that pays a Brazilian company, that pays a Russian company to do the actual residential scraping? The same example could be made with money laundering, and yes it's a…

Your money laundering analogy is apt. I know very little about that topic, and I especially don't know how much money laundering is really out there (nor do governments), but I'm confident that a lot is. Do AML laws have a chilling effect on it? I think they must, since they surely increase the cost and risk, and similar legislation for scraping should have a similar effect. But AML is a pretty bad solution to money laundering, and I despair if AML-for-scraping is the best possible solution to scraping.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#479

Post by BrenBarn »

akoboldfrying wrote: post #477
BrenBarn wrote: post #463

Earlier quoted context omitted.

Using "open and free" to mean "I actually want no restrictions at all" is also confusing and disingenuous, because, as you yourself point out, a lot of people don't mean that by those words. The other thing, though, is that there's a difference between "I personally want to release my personal work under open, free, and unrestricted terms" and "I want to release my work into a system that allows people to access info…

> Asking people to just unilaterally disarm by imposing no restrictions I'm not asking for this. I'm asking for people who want such restrictions (most of which I consider entirely reasonable) to say so explicitly. It would be enough to replace words like "free" or "open" with "fair use", which immediately signals that some restrictions are intended, without getting bogged down in details.

Why? It seems you already know what people mean by "open and free", and it does have a connection to the ideals of openness and freedom, namely in the systemic context that I described above. So why bother about the terminology?

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#480

Post by AnthonyMouse »

Lammy wrote: post #288
AnthonyMouse wrote: post #273

Earlier quoted context omitted.

You can still do that, you have the caching reverse proxy at the edge of the network be the thing that terminates TLS.

Not really. At minimum you will break all of these sites on the HSTS preload list: https://source.chromium.org/chromium/chromium/src/+/main:net...

It isn't the client side who does this, it's the server side. Doing it on the client side has a nominal benefit in the typical case but is very little value to you when the problem is some misbehaving third party AI scraper taking down the server when you need to get something from it that isn't already in the local cache.

If you have three local machines, you might be able to turn three queries into one, assuming they all visit the same site instead of different people using different sites.

If you do this on the server, a request that requires the execution of PHP code and three SQL queries goes from happening on every request for the same resource to happening once and then the subsequent requests are just shoveling the cached response back out the pipe instead of having to process it again. Instead of reducing the number of requests that reach the back end by 3:1 you reduce it by a million to one.

And that doesn't cause any HSTS problems because a reverse proxy operated by the site owner has the real certificate in it.

Post reply on HN