What happens when Google starts using the data scrapped by its search engine crawlers for AI training? What prevents another crawler from impersonating one that is part of this program and getting someone else to pay? What happens when people start using headless browsers as crawlers and they are undetectable?
Cloudflare to introduce pay-per-crawl for AI bots
181–190 of 317 posts
Re: Cloudflare to introduce pay-per-crawl for AI bots
#182Earlier quoted context omitted.
The cynic in me says we'll be seeing articles about blog owners getting fractions of a tenth of a penny while Cloudflare pockets most of the revenue. And of course it will eventually be rolled out for everyone, meaning there will be a Cloudflare-Net (where you only can read if you give Cloudflare your credit card number), and then successively more competing infrastructure services (Akamai, AWS, ... meaning we get in…
I worked at Cloudflare for 3 years until very recently, and it's simply not the culture to behave in the way that you are describing. There exists a strong sense of doing the thing that is healthiest for the Internet over what is the most profit-extractive, even when the cost may be high to do so or incentives great to choose otherwise. This is true for work I've been involved with as well as seeing the decisions mad…
You are probably right that this is not the case right now. 25 years ago you could say the same about google employees. Incentives change with time, and once infrastructure is in place it's nearly impossible to get rid of it again.
So one better makes sure that it has not the potential to further introduce gatekeepers, where later such gatekeepers will realize that, in order to continue to live, they need to make a profit over everything else, and then everything is out of the window.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#183Seems like Google got a pass, since their search engine activity is not included.
Once they cached the page, their AI can "crawl" internally.
(Same applies to Bing most likely)
Re: Cloudflare to introduce pay-per-crawl for AI bots
#184This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…
As a Startup I absolutely want to get crawled. If people ask ChatGPT "Who is $CompanyName" I want it to give a good answer that reflects our main USPs and talking points. A lot of classic SEO content also makes great AI fodder. When I ask AI tools to search the web to give me a pro/con list of tools for a specific task the sources often end up being articles like "top 10 tools for X" written by one of the companies o…
Most people are not startup owners
Re: Cloudflare to introduce pay-per-crawl for AI bots
#185Earlier quoted context omitted.
Because Visa doesn't (hasn't) wanted to do microtransactions, ever, since the beginning of the internet.
You still don't need cryptocurrencies. You just need a middleman that aggregates micropayments into large enough amounts to work with non-micropayment systems. Some might object to having to get middlemen involved, but the thing is that even with cryptocurrency payments you are going to need middlemen because the web is international. If your website is directly charging crawlers to crawl and you get crawled and paid…
A middleman is not strictly required for cryptocurrencies. Regulations around them and how international transactions are taxed will depend on each country, just like anything else. These matters can be handled by lawyers and accountants as usual.
While I agree that cryptocurrencies are not strictly required for this, the infrastructure already exists to support micropayments, and is well understood and trusted. What infrastructure could support the same use cases for fiat micropayments? Would it be as low friction to setup and use as cryptocurrencies are today? Would it be decentralized and not depend on a single company?
I'm as tired as anyone else about the cryptocurrency hype and the charlatans and scammers it has enabled. But I also think it's silly to completely ignore the technology and refuse to acknowledge that it has genuine use cases that no other system is well suited for. Micropayments and powering novel business models on the web is one clear example of that.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#186Earlier quoted context omitted.
What if my phone rings and I forget to close the page? Or leave it open to read it later? Plus as one of the parent comments said, I am not paying before I get an idea what I'm paying for.
> Plus as one of the parent comments said, I am not paying before I get an idea what I'm paying for. The logic was, you aren't paying to a website. You're paying to a broker that distributes how much you've paid to all websites you've visited that have opted in, prorated based on how much time you spent on them.
> You're paying to a broker that distributes how much you've paid to all websites you've visited that have opted in
That does make sense. Although that broker is going to push for a subscription.
> prorated based on how much time you spent on them
If it's text, I wouldn't mind, I'm a very fast reader. But that's incentive to lengthen the "content" or maybe go all video, insert animated page transitions and stuff like that.
Slow readers and people on slow internet will be penalized.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#187This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402
Maybe I'm wrong, I hope I am, but it feels like the boats out for micro payments. To me at least, it feels like for this system to work you want to have something like what PAYG phones have with top-ups. You "put a tenner on your internet", and sites use that in the form of micro payments. Had that been the case since the start, it could've worked great, but now the amount of infrastructure and buy-in required to make that work, it just feels like we missed the chance.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#188The more I read this, the more I feel like web attestation is going to be suggested as a way to prove AI bots vs humans.
They've been trying to push this through for a while now (to some moderate success). This may be the final push they are looking for to get it more thoroughly integrated in the web as a whole.
I know it wasn't mentioned anywhere here, but it's the silent part that fits this puzzle piece really well.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#189Earlier quoted context omitted.
This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…
Even if advertising were to disappear over night, why do you think that would stop the spread of propaganda, corruption of democratic processes, and social unrest? I don't really see a connection between the two?
Where did I say that all of those things would stop?
What I said is that adtech systems are also used for it. So if they were to disappear overnight, a _proportion_ of those activities, and a pretty large one I reckon, would also disappear.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#190Earlier quoted context omitted.
If you don't want payment, there is: https://anubis.techaro.lol/ Used by https://gcc.gnu.org/bugzilla/ for example. It is less annoying than CAPTCHA/Turnstile/whatever because the proof of work runs automatically.
Sadly they seem to be getting through this one lately. Had a scraper hitting me with 80 qps punching straight through Anubis. Had to set up a global rate limit that browned out the functionality they were interested in[1] in excessive load. [1] This form https://marginalia-search.com/site/news.ycombinator.com
Probably headless chrome. I'm going to investigate.