Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

241–250 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#241

Post by sc68cal »

adjfasn47573 wrote: post #218

I see most people stating that the internet as we know it could be gone because of AI. I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today? There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for…

> An internet with less ads, more genuine information. If AI will lead to this next phase of the internet

How is AI supposed to create an internet "with more genuine information", based on what we have seen so far? These two statements appear to be mutually exclusive.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#242

Post by __MatrixMan__ »

BlueTemplar wrote: post #237
__MatrixMan__ wrote: post #231

Earlier quoted context omitted.

I disagree, bundling is the problem. That strategy created the fragmented landscape that we now see in streaming video, which is pretty much universally hated. The ideal solution would involve a flat rate which I pay monthly, and at the end of the month that money goes towards the content that I consumed during that month. If I only read a single blog, they get all of it. Then we build a culture around preferring to…

So, Flattr 2.0 ? https://www.ctrl.blog/entry/flattr2.html

More or less, yeah.

When it's all grown up though, I'd hope for more transparency into where the money is going. Suppose a journalist has risked life and limb to expose some important information and two news outlets publish stories about it. I don't want to pay the news outlets under the assumption that they'll then pay the journalist. Instead I want to decide which story to read based on whichever one triggers my client to compensate the journalist the most (because I care more about the investigative work than the writing, though other users might configure their clients differently).

Re: Cloudflare to introduce pay-per-crawl for AI bots

#243

Post by ASalazarMX »

sc68cal wrote: post #241
adjfasn47573 wrote: post #218

I see most people stating that the internet as we know it could be gone because of AI. I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today? There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for…

> An internet with less ads, more genuine information. If AI will lead to this next phase of the internet How is AI supposed to create an internet "with more genuine information", based on what we have seen so far? These two statements appear to be mutually exclusive.

If I understand correctly, it will be not by creating a new iteration, but by destroying the current one.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#244

Post by johnklos »

lblume wrote: post #36
JimDabell wrote: post #33

This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost. Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastru…

But don't these new costs create a direct incentive to cooperate?

No. Companies don't care about saving money by itself. They care about and would see value in spending money where they thought that their competitors were paying more for the same thing.

It's similar to this fortune(6):

    It is not enough to succeed.  Others must fail.
      -- Gore Vidal

Re: Cloudflare to introduce pay-per-crawl for AI bots

#245

Post by dogleash »

adjfasn47573 wrote: post #218

I see most people stating that the internet as we know it could be gone because of AI. I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today? There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for…

I agree with the premise about impermanence. But moving in the direction of "less ads, more genuine" is comical if not tied to the userbase completely falling out and most never coming back.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#246

Post by kentonv »

SparkyMcUnicorn wrote: post #209
kentonv wrote: post #126

Earlier quoted context omitted.

How would an individual user use a "crawler" to navigate the web exactly? A browser that uses AI is not automatically a "crawler"... a "crawler" is something that mass harvests entire web sites to store for later processing...

How can you tell the difference, in a way that can't be spoofed? This is a genuine question, since I see you work at CF. I'm very curious what the distinction will be between a user and a crawler. Is trust involved, making spoofing a non-issue?

I don't personally work on bot detection, and I don't know exactly what techniques they use.

But if you think about it: crawlers are probably not hard to identify, as they systematically download your entire web site as well as every other site on the internet (a significant fraction of which is on Cloudflare). This traffic pattern is obviously going to look extremely different from a web browser operated by a human. Honestly, this is probably one of the easiest kinds of bots to detect.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#247

Post by MisterTea »

jorvi wrote: post #227
c4wrd wrote: post #226

Earlier quoted context omitted.

You're missing the bigger picture. It isn't free to put content on the Internet. At a bare minimum, you have infrastructure and bandwidth costs. In many cases, a goal someone may have is that if they publish content on the internet, they will attract people to return for more of the content they produce. Google acted as a broker, helping facilitate interactions between producers and consumers. Consumers would supply…

> A clearer picture to help you understand what's going on: the internet of the past few decades was a bazaar marketplace. That internet died almost two decades ago. Not sure what you're talking about.

The web died. The internet is still a functional global IP network. For now.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#248

Post by johnklos »

PeterStuer wrote: post #212
9283409232 wrote: post #122

Earlier quoted context omitted.

I don't trust Cloudflare but this is not a problem they created. They solved a real problem with DDoS protection in the beginning and now AI crawlers increasing server cost is not a negligible problem. The CEO of iFixit called out Anthropic publicly for hitting their site a million times in 24 hours to scrape it. We are passed the point of good faith action from these AI companies. They are adversarial and need to be…

But they do create this problem. They could have specifically defaulted to not blanketing RSS feeds and other M2M specific pages. Instead, we are now in a situation where even daring to look at robots.txt can flag you as a bot.

True. For all those people who want to make excuses for Cloudflare, it's an excellent reminder that they've known about this problem for years and they still haven't fixed it.

Are they inept? Or do they really only care about things that bring them profit and that normalize their marginalization of non-paying groups? Which explanation makes the most sense?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#249

Post by __MatrixMan__ »

eddythompson80 wrote: post #240
__MatrixMan__ wrote: post #231

Earlier quoted context omitted.

I disagree, bundling is the problem. That strategy created the fragmented landscape that we now see in streaming video, which is pretty much universally hated. The ideal solution would involve a flat rate which I pay monthly, and at the end of the month that money goes towards the content that I consumed during that month. If I only read a single blog, they get all of it. Then we build a culture around preferring to…

It's not an ideal solution because any fixed cost solution is begging for a middle man/reseller to be introduced. Like I pay the $5 monthly flat fee (or $500, $5k, $500k, whatever it's known fixed cost for me) for the system, turn around and resell all content for a $1 monthly flat fee. There is a real cost to the content you're consuming with that flat-fee. So either the flat fee is more of "credit" system or it's r…

I suppose "credit system" is indeed more accurate than "fee", it's just that I personally would set it at a flat rate and then stop thinking about it, so it would feel like a sort of admission-to-the-internet to me.

As for bandwidth and storage costs... that could just be rolled into the same attribution/payment scheme. If content is not propagating well because too few people are hosting it, then I'm ok with allocating some space and bandwidth to help distribute it. I don't think there's anything wrong with that so long as when it gets viewed, the creators still get the bulk of the credit and I only get a teensy bit for the part I played in distributing it.

The goal would be to mostly decouple the attribution/payment handling from the data handling so that it's as simple as seeding a torrent and it's the players/clients/whatever that handles giving credit. If I notice that I've got a leacher problem (whether as a creator or as a distributor) then maybe I revoke trust in the leachers and they stop getting the content from me.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#250

Post by chrisweekly »

blackjack_ wrote: post #220
squigz wrote: post #196

Earlier quoted context omitted.

Okay, but my question remains - why? What is the connection between those things and advertising? It seems way more likely to me that they would simply adapt, as they always have.

Sure, I can explain. Basically, adtech is the backbone of the attention economy where more clicks = more revenue. So the incentives are to always say the most inflammatory clickbait you can, to incentivize profits. Sensible and boring stable takes and agreement will always be stifled to promote outrage, beefs, and clickbait to maximize revenue. To generalize; stability in any general field like politics or journalism…

Well put.

Also, (tangent), I misread "stable takes" as "table stakes", having never seen that phrasing for the opposite of "hot takes". I like it.

Post reply on HN