I'm not challenging that's how they work, I
also understand how they work, perhaps not on a technical nuts-and-bolts way, but in general way enough to critique it. That is, in fact, my critique and why I hate these tools so much: no matter how many guardrails you put in, or how much filtering, or how much oversight by another goddamn LLM or five or whatever, that doesn't solve the issue.
You have with these things something that resembles at least, a black box of a reasoning machine. I'm not going to litigate how much or how little, whatever, we'll just hand-wave that part away. The problem remains the same: that if anything, ANYTHING at all, in the training data points at something inappropriate, that inappropriate thing is now accessible. And it was clear from the jump with widespread scraping of data from all corners of the internet that there would be huge amounts of inappropriate material of ALL kinds in those datasets, and it's only become more clear with more time with these tools, and seeing what people can make them do.
And thus far, the AI industry's only answer is bolting on, as stated elsewhere, other systems to check the prompts before they go in, and/or review the outputs before they are sent to users. And it is also clear that these systems are just as imperfect as the thing you are trying to guardrail in the first place!
And exactly what I and many others predicted, and why we said "please don't build this" for YEARS, has happened. We've gotten literally everything: they'll generate stuff that violates copyright, they will regurgitate items directly from training data and present it as new, they will make shit up wholesale, they will generate nudes of people without consent, on, and on, I cannot stress enough that every single nightmare scenario attributed to this tech has been found, presented, reproduced, and the vast majority are still eminently possible to do via established, frontier products by the largest vendors in the space.
This. Is. Ridiculous.
I get the impression from the tone of your message that you are either pro-AI or perhaps work on AI, and I get that nobody likes being criticized. But COME ON. We have been at this for over three years! The people behind this tech have been trying to build the torment nexus and have largely succeeded, and every time that gets pointed out, we have to listen to people go "well it's not thaaaat bad"
Yes it is. Yes it fucking is. It is bad for IP owners, it's bad for users, it's bad for UX, it's bad for the environment, it's bad for the PC market, it's bad for software engineers, it's bad for education, it's bad for hiring, it's bad for hollywood, it's bad for marketing. The ONLY people who like this shit are business weirdos and middle managers. And nvidia.