> What problem does this solve that a basic API key doesn't solve already?
Many things, including improved security, and the possibility of delegating authorization in the ways described in their article (if you do not restrict the certificate from issuing further certificates, and if you define an extension for use with your service to specify narrower authorization, and document this).
> The issue with that approach is that you will require accounts/keys/certificates for all hosts you intend to visit, and malicious bots can create as many accounts as they need. You're just adding a registration step to the crawling process.
Read the last paragraph of what I wrote, which explains why that issue does not apply. However, even if registration is required (which I say should not be required for most things anyways, especially read-only stuff), it does not necessarily have to be that fast or automatic.
> Your suggested approach works for websites that want to offer AI access as a service to their customers, but the problem Cloudflare is trying to solve is that most AI bots are doing things that website owners don't want them to do. The goal is to identify and block bad actors, not to make things easier for good actors.
The approach I describe would work for many things where authentication and authorization helps (most of which does not involve AI).
I do know that it does not solve the problem that Cloudflare is trying to solve, but it does what it says in the article about authorization, and in a secure way. And, it is open, interoperable, and standardized.
The problem that Cloudflare is trying to solve cannot be solved in this way, and the way Cloudflare tries to do it is not good either.
Things that AI bots are doing to other's sites includes such things as excessive scraping, rather than accessing private data (even if they might do that too, Cloudflare's solution won't help with that at all either). (There is also excessive blocking, but Cloudflare is a part of the problem, even if some of the things they do sometimes help.)
See comment 45068556. Not everything should require authentication or authorization. Also see many other comments, that also mention why it does not help.
> Using mTLS/client certificates also exposes people (that don't use AI bots) to the awful UI that browsers have for this kind of authentication. We'll need to get that sorted before an X509-based solution makes any sense.
OK, it is a valid point, but this could be improved, independently. (Before it is fixed (and even afterward if wanted), X.509 could be made as only one type of authentication; the service could allow using a username/password (and/or other things, such as TOTP) as well for people who do not want to use X.509.)
Also, AI bots are not the only kind of automated access (and is not one that I use personally, although other people might); you could also be using a API for other purposes, or you might be using a command-line program for manual access without the use of a web browser, etc.