Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

301–310 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#301

Post by rickdeckard »

dhx wrote: post #298
rickdeckard wrote: post #141

Earlier quoted context omitted.

> It depends on the content producer. I would argue the best resourced content producers (governments and large companies) are incentivised to give AI bots as much curated content as possible that is favourable to their branding and objectives. Yeah, if the content being processed is NOT the product being sold by the creator. > [..] the report makes fairly clear some of the BBC's motivation for global reach that shou…

> What kind of monetization model would this be for BBC? BBC is taxpayer funded by Britons who have elected British officials convinced of the benefits of using taxpayer funds to influence perception (or even just prevent misinformation) of Britain and Britons, using BBC's global reach. That is, BBC is not required to completely balance the books with consumer payments received in exchange for entertainment. British…

You're mixing BBC the UK broadcaster with BBC Studios, the internationally acting commercial (!) subsidiary of BBC.

The latter made 1.8bn GBP revenue in 2024

Re: Cloudflare to introduce pay-per-crawl for AI bots

#302

Post by onesociety2022 »

davkan wrote: post #287
ashdksnndck wrote: post #285

Earlier quoted context omitted.

The whole point of bundling is that you combine many things together because the marginal cost is zero. If you pay for YouTube premium, you are also paying for all of the videos on YouTube that don’t interest you. That’s irrelevant as long is there is enough content you want. The fact that the bundle includes a music app you don’t care to use is the same way, it doesn’t detract from the value proposition.

Well in my case the value is not there at their price which includes YouTube music as a separate service that must be maintained and I assume separate licensing as well. I’m not interested in paying for access to that side of the service. I have little doubt that the access to YouTube music adds to the sticker price of their bundle. They are not simply passing along bundled costs, they are also targeting what they th…

I normally would subscribe to Apple Music, but I switched to Youtube Premium because I get Youtube Music with the added benefit of no ads on regular Youtube. It seems like their bundling trick worked on me as they expected.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#303

Post by onesociety2022 »

__MatrixMan__ wrote: post #233
gessha wrote: post #120

Earlier quoted context omitted.

But why exactly does it have to be on an append-only ledger where transactions are processed/validated for a fee? Why can’t it be a more conventional transaction processor like VISA on top of the banking system?

Because conventional transaction processors can be compelled to shut off payments to publishers whose content offends the powerful. Just look at what happened to wikileaks.

Crypto is not going to save you there. If some country like India wants to ban content from some publishers, they can do so by just asking their ISPs to ban all traffic to said publishers. Whether the publisher accepts micropayments on crypto rails or conventional rails is immaterial. They are not going to be able to distribute their content anymore and hence not get paid either.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#304

Post by davkan »

onesociety2022 wrote: post #302
davkan wrote: post #287

Earlier quoted context omitted.

Well in my case the value is not there at their price which includes YouTube music as a separate service that must be maintained and I assume separate licensing as well. I’m not interested in paying for access to that side of the service. I have little doubt that the access to YouTube music adds to the sticker price of their bundle. They are not simply passing along bundled costs, they are also targeting what they th…

I normally would subscribe to Apple Music, but I switched to Youtube Premium because I get Youtube Music with the added benefit of no ads on regular Youtube. It seems like their bundling trick worked on me as they expected.

I wonder if that makes up for the money they are losing as I continue to abuse their service with ad blockers and sideloading modified ad free clients.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#305

Post by Mars008 »

chomp wrote: post #184
wongarsu wrote: post #147

Earlier quoted context omitted.

As a Startup I absolutely want to get crawled. If people ask ChatGPT "Who is $CompanyName" I want it to give a good answer that reflects our main USPs and talking points. A lot of classic SEO content also makes great AI fodder. When I ask AI tools to search the web to give me a pro/con list of tools for a specific task the sources often end up being articles like "top 10 tools for X" written by one of the companies o…

Most people are not startup owners

I'm afraid google's search and AI bots are the same. Either get both or none. None means no visibility in web. Some will be fine with this, but most will agree on AI bots to get visibility. Soon AI providers will charge those who want to be included, if they don't do it yet. With this on request they'll return preferred products and services. Just another form of hidden advertising.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#309

Post by KETpXDDzR »

Can't (AI) crawlers just use a browser user agent? TL;DR: Yes, but with some more work that the big AI companies can surely manage to implement.

    To bypass detection, a crawler would need to mimic not just the user agent, but also the full set of browser behaviors, headers, and environmental fingerprints—a much more complex task that Cloudflare’s systems are designed to counter.
Post reply on HN