Cloudflare to introduce pay-per-crawl for AI bots

This is really interesting. Assuming I understood it correctly, I wonder why the protocol does not allow immediate return when it gave an address and payment amount. Subsequent attempts should be blocked until some kind of checksum of amount and wallet address is returned. This checksum should be verified by a third-party. This would save each server from implementing the verification logic.

Two missing pieces that would really help build a proper digital economy are:

1. If the content could be consumed by only the requesting party, and not copied and stored for future,

2. if there is some kind of rating on the content, ideally issued by a human.

Maybe some kind of DRM or Homomorphic Encryption could solve the first problem and the second could be solved by human raters forming DAO based rating agencies for different domains. Their expertise could be gauged by blockchain-based evidences and they will have to stake some kind of expensive cryptocurrency to join such a DAO akin to license. Content and Raters could be discovered via like BitTorrent Indexes, thus eliminating advertisers.

I say these as missing pieces because it will allow humans to remain an important part of digital economy by supplying their expertise, while eliminating the middle man. Humans should not be simply cogs in digital economy whose value are extracted and then discarded but should be the reason for its value.

By solving double-spending problem on content we ensure that humans are paid each time. This will encourage them to keep on building new expertise in offline ways - thus advancing civilization.

For example when we want a good book to read or movie to watch, we look at Amazon ratings or Goodreads review. The people who provide these ratings have little skin in the game. If they have to obtain license and are paid, then when they rate an authorship - just like bonds are rated by Rating agencies - the work can be more valuable. Everyone will have reputation to preserve.

Especially when it asks you to do it again, and again, until you realise you're never going to be allowed to see the page.

I've had that happen a few times with sites behind Cloudflare.

My gut reactions…

- I agree that something like this is necessary or the whole model of the internet will be broken, like Matthew Prince [explained in this video](https://www.youtube.com/watch?v=H5C9EL3C82Y).

- Their approach seems very imperfect, but I understand that you have to start somewhere.

- They are paying per crawl… but in fairness it should really be per usage. It’s like paying music artists once when they upload to Spotify rather than per-play -- even though one artist gets zero plays and another gets ten million. Sure, the idea is crawlers will bid more for the popular content author, but what if a nobody author has a one-hit-wonder piece of content. They’ll still just get a couple bips per crawl and then the cat is out of the bag.

- One solution to this would be requiring a GDPR-style forget mechanism, where the author is granting a limited-duration license for the content (say… one week), after which it must be deleted and re-licensed. This would be a huge fix for the whole thing… and the more I think about it the more I think it’s essential for this to work.

- The auction mechanics are biased to the crawler… if there is a spread between artist price and crawler max price, then the crawler pays the lower price set by the artist. It should be the average.

- They will need to provide content authors with analytics about the pricing mechanics for the bids the crawlers are making.

- If this whole thing works, then products that optimize bid mechanics on behalf of authors will be a big growth industry.

- If Cloudflare are setting themselves up as the clearing mechanism for payments, that’s far too much power and profit for one company. It’s even worse than the Google monopoly. Somehow the payment mechanics need to be democratized.

And then 20 years later Cloudflare hits hard times and gets bought by someone you don't like. The problem is that much power concentrated in any one place.

How can you tell the difference, in a way that can't be spoofed?

This is a genuine question, since I see you work at CF. I'm very curious what the distinction will be between a user and a crawler. Is trust involved, making spoofing a non-issue?