This argument is weak. If we agree that the production is infringing, then selling a machine that produces infringing works on demand is also infringing to the rights of the author. For example, if I sell a karaoke machine that comes with pre-recorded famous melodies without the original vocals (thus, derived works), I definitely cannot claim "only the agency of the users matter". No, even the on-demand production itself is an act of infringement.
Regarding your fair use point:
> add something new, with a further purpose or different character, and do not substitute for the original use of the work.
This is a test that can be applied to the works produced, and I think that, by and large, it applies to most of them, they are sufficiently transformative to be seen as fair use, especially since they are produced and published, as you say, under the agency of a human user of the system.
But we're talking here about the system as a whole, not a certain production. The ChatGPT model database is not a creative work (other that its source code), it's a binary compressed representation of the works used to train it, for the purpose of on-demand production of works that mimicry, to a varying, and sometimes strikingly infringing degree, the originals. There is no question such a machine competes with the originals in the market place, there is no real question it is a derivative work of the originals, that fully supply its artistic features and without which the machine could not utter a single intelligible sentence.
Also, the "one word out of 1000" analogy defense does not apply, since a single isolated word is not a work of creation, while a certain shape, brush stroke or cromatic set are. AI does not create using disparate pixels of the originals, it uses entire creative features, more like publishing a book where each page or chapter is copied verbatim and adapted from other works, or if characters, plot devices and story structure are reused. These latter cases have long been recognized as plagiarism.
So the legal question seems to be if mass-scale algorithmic remixing and derivation by a non-creative competitor is fair use. That strikes me as far outside the intended spirit of fair use and the balance it seeks to achieve between author protection and beneficial social progress.