Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

221–230 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#221
post #121
post #46

Earlier quoted context omitted.

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…

Even if advertising were to disappear over night, why do you think that would stop the spread of propaganda, corruption of democratic processes, and social unrest? I don't really see a connection between the two?

The finiancial connexion is explained here:

https://en.wikipedia.org/wiki/Citizens_United_v._FEC

Re: Cloudflare to introduce pay-per-crawl for AI bots

#222
post #200
post #196

Earlier quoted context omitted.

Okay, but my question remains - why? What is the connection between those things and advertising? It seems way more likely to me that they would simply adapt, as they always have.

The connection is that those wishing to influence public opinion can do so by running ad campaigns that target precisely the demographic they wish to manipulate. Adtech doesn't care whether you're promoting products or ideas. This connection should be obvious after the Cambridge Analytica leak. Social media and any media platform also enables the spreading of propaganda, but it's not as systematic as the tools built…

In support of your position:

https://en.wikipedia.org/wiki/Citizens_United_v._FEC

Re: Cloudflare to introduce pay-per-crawl for AI bots

#223

I see most people stating that the internet as we know it could be gone because of AI. I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today? There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for…

> all sorts of people trying to exploit it for money

Because they could. In AI-first web, people can't really do anything about anything - only those in control of training the handful of "big popular AI models" are the gatekeepers of all knowledge.

> with less ads, more genuine information

That's orthogonal to AI. Models are already being trained to favour certain products/services and they already (re)produce factually incorrect information with no way to verify or correct them.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#224
There’s a tiny but important difference between scraping and getting work done.

Scraping is associate with mindless extraction. Like a vacuum cleaner sucking-in any data without context, permission, or contributing value back.

On the other hand AI agents aren’t here to scrape for the sake of it - I have seen it first hand. They are here to get work done, mostly researching, summarizing, assisting, building new products. You could argue this data is then use to train further a model, you would probably argue correctly, but that’s a topic for another day.

I implemented the poor man's version demo of what a similar concept to this could be like: http://github.com/toolhouseai/fastlane-demo

Re: Cloudflare to introduce pay-per-crawl for AI bots

#225
This is cool but I don't like how this forces all crawlers to use Cloudflare. Google Chrome developers were proposing some Web Monetization API in Chromium a few years back, back when the Manifest V3 drama was still fresh, so maybe we should look into that instead. To allow decentralized payments to not be dependent on a single vendor.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#226

I see most people stating that the internet as we know it could be gone because of AI. I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today? There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for…

You're missing the bigger picture. It isn't free to put content on the Internet. At a bare minimum, you have infrastructure and bandwidth costs. In many cases, a goal someone may have is that if they publish content on the internet, they will attract people to return for more of the content they produce. Google acted as a broker, helping facilitate interactions between producers and consumers. Consumers would supply a query they want an answer to, and a producer would provide an answer or facilitate a space for the answers to be found (in the recent era, replace answer with product or store-front).

There was a mostly healthy interaction between the producers and consumers (I won't die on this hill; I understand the challenges of SEO optimization and an advertisement-laden internet). With AI, Google is taking on the roles of both broker and provider. It aims to collect everyone's data and use it as its own authoritative answer without any attribution to the source (or traffic back to the original source at all!).

In this new model, I am not incentivized to produce content on the internet, I am incentivized to simply sell my data to Google (or other centralized AI company) and that's it.

A clearer picture to help you understand what's going on: the internet of the past few decades was a bazaar marketplace. Every corner featured different shops with distinct artistic styles, showcasing a great deal of diversity. It was teeming with life. If you managed your storefront well, people would come back and you could grow. In this new era, we are moving to a centralized, top-down enterprise. Diversity of content and so many other important attributes (ethos, innovation, aestheticism) go out of the window.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#227
post #226

I see most people stating that the internet as we know it could be gone because of AI. I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today? There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for…

You're missing the bigger picture. It isn't free to put content on the Internet. At a bare minimum, you have infrastructure and bandwidth costs. In many cases, a goal someone may have is that if they publish content on the internet, they will attract people to return for more of the content they produce. Google acted as a broker, helping facilitate interactions between producers and consumers. Consumers would supply…

> A clearer picture to help you understand what's going on: the internet of the past few decades was a bazaar marketplace.

That internet died almost two decades ago. Not sure what you're talking about.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#228
post #107

This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost. Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastru…

>Crawling the web is not a competitive advantage for any of these AI companies, ?? it's their ability to provide more up to date information, ingest specific sources, so it is definitely a competitive advantage to have up to date information them not paying the content of the sites they index and read out, and not referring anybody to their sites is what will kill the web as we know it. for a website owner there is z…

> for a website owner there is zero value of having their content indexed by AI bots. Zilch.

This very much depends on how the site owner makes money. If you’re a journalist or writer it’s an existential threat because not only does it deprive you of revenue but the companies are actively trying to make your job disappear. This is not true of other companies who sell things other than ads (e.g. Toyota and Microsoft would be tickled pink to have AI crawl them more if it meant that bots told their users that those products were better than Ford and Apple’s) and governments around the world would similarly love to have their political views presented favorably by ostensibly neutral AI services.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#229
post #94

So we used to have this company that did good things for the internet... like usable search... Now we have this company that does good things for the internet... like ddos protection, cdns, and now protecting us from "AI"... How long will the second one last before it also becomes universally hated?

Good things for the Internet? I stop visiting sites that nag me with their verification friction. They are the only reason I replaced Stack Exchange with LLMs.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#230
post #226

I see most people stating that the internet as we know it could be gone because of AI. I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today? There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for…

You're missing the bigger picture. It isn't free to put content on the Internet. At a bare minimum, you have infrastructure and bandwidth costs. In many cases, a goal someone may have is that if they publish content on the internet, they will attract people to return for more of the content they produce. Google acted as a broker, helping facilitate interactions between producers and consumers. Consumers would supply…

> You're missing the bigger picture. It isn't free to put content on the Internet. At a bare minimum, you have infrastructure and bandwidth costs.

While it technically isn't free, the cost is virtually zero for text and low-volume images these days. I run a few different websites for literally $0.

(Video and high-volume images are another story of course)

Post reply on HN