So we used to have this company that did good things for the internet... like usable search... Now we have this company that does good things for the internet... like ddos protection, cdns, and now protecting us from "AI"... How long will the second one last before it also becomes universally hated?
Cloudflare to introduce pay-per-crawl for AI bots
211–220 of 317 posts
Re: Cloudflare to introduce pay-per-crawl for AI bots
#212So: 1. Encourage fencing off everything by default to maximize need for bypass 2. Offer bypass through payment 3. Profit! You wouldn't believe the number of public administrations with public information that have (mostly unwittingly) had some lazy contractor put Cloudflare in front of their entire site, blocking even their RSS feeds from M2M. Yes, you can send them mails and call and sometimes, if they even understa…
I don't trust Cloudflare but this is not a problem they created. They solved a real problem with DDoS protection in the beginning and now AI crawlers increasing server cost is not a negligible problem. The CEO of iFixit called out Anthropic publicly for hitting their site a million times in 24 hours to scrape it. We are passed the point of good faith action from these AI companies. They are adversarial and need to be…
Re: Cloudflare to introduce pay-per-crawl for AI bots
#213This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402
As someone who has actually built working micro payments systems, this was of interest. Worth noting though that it's really just "document-ware" -- there's no code there[1], and their proposed protocol doesn't look like it was thought through to the point where it has all the pieces that would be needed. [1] E.g. this file is empty: https://github.com/coinbase/x402/blob/main/package.json
The Github repo clearly has Python and Typescript examples of both client and server (and in multiple frameworks), along with Go and Java reference implementations.
Maybe check the whole repo before calling something vaporware?
Re: Cloudflare to introduce pay-per-crawl for AI bots
#214Earlier quoted context omitted.
The cynic in me says we'll be seeing articles about blog owners getting fractions of a tenth of a penny while Cloudflare pockets most of the revenue. And of course it will eventually be rolled out for everyone, meaning there will be a Cloudflare-Net (where you only can read if you give Cloudflare your credit card number), and then successively more competing infrastructure services (Akamai, AWS, ... meaning we get in…
I worked at Cloudflare for 3 years until very recently, and it's simply not the culture to behave in the way that you are describing. There exists a strong sense of doing the thing that is healthiest for the Internet over what is the most profit-extractive, even when the cost may be high to do so or incentives great to choose otherwise. This is true for work I've been involved with as well as seeing the decisions mad…
Re: Cloudflare to introduce pay-per-crawl for AI bots
#215meh. ads for ai content is really the answer - e.g "OpenAI ads" content creator puts a tag on their page / set their domain - when the crawler sees it, display an ad pass on $ as usual.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#216This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402
This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…
When someone on the internet tries to sell you something for a dollar, how often do you really take them up on it? How many microtransactions have you actually made? To problem with microtransactions is they discourage people from consuming your content. Which is silly, because the marginal cost of serving one reader or viewer is nearly zero.
The solution is bundling. I make a decision to pay once, then don’t pay any marginal costs on each bit of content. Revenue goes to creators proportionally based on what fraction of each user’s consumption went to them.
People feel hesitation toward paying for the bundle, but they only have to get over the hump once, not repeatedly for every single view.
Advertising-supported content is one kind of bundle, but in my opinion, it’s just as exhausting. The best version of bundling I’ve experienced are services like Spotify and YouTube Premium, where I pay a reasonable fixed monthly fee and in return get to consume many hours of entertainment. The main problems with those services are the middlemen who take half the money.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#217Earlier quoted context omitted.
Much cheaper than using a credit card processor.
Not cheaper than using one of the instant net settlement services like FedNow (UPI in India, PromptPay in Thailand, PayID in Australia)
Re: Cloudflare to introduce pay-per-crawl for AI bots
#218I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today?
There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for money.
I welcome a change. An internet with less ads, more genuine information. If AI will lead to this next phase of the internet, so be it. And this phase won’t be the last either.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#219What about if somebody uses artificial intelligence crawler to help them navigate the web as an accessibility tool? Enabling UI automation. It already throws up a lot of... uh... troublesome verifications.
We already have ARIA, which is far more deterministic and should already be present on all major sites. AI should not be used, or necessary, as an accessibility tool.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#220Earlier quoted context omitted.
Where did I say that all of those things would stop? What I said is that adtech systems are also used for it. So if they were to disappear overnight, a _proportion_ of those activities, and a pretty large one I reckon, would also disappear.
Okay, but my question remains - why? What is the connection between those things and advertising? It seems way more likely to me that they would simply adapt, as they always have.
Basically, adtech is the backbone of the attention economy where more clicks = more revenue. So the incentives are to always say the most inflammatory clickbait you can, to incentivize profits. Sensible and boring stable takes and agreement will always be stifled to promote outrage, beefs, and clickbait to maximize revenue. To generalize; stability in any general field like politics or journalism gets turned into obnoxious grandstanding to be more like reality tv to get more attention. In software, people who monetize off advertising are incentivized to build dark patterns maximized on attention grabbing. Whereas without advertising as the main source of revenue, people stop building dark these patterns to steal your attention, as you are paying them directly for a service, so you are the customer instead of the product.