> I can understand that people believe this, but why do they do? Nothing in our society operates in a way that might imply this.
Sure, but that disconnect between what people think and how things work is almost fully general over all subjects.
I've seen people (behave as if they) think translation is just the words, but that leads to "hydraulic ram" becoming "water sheep". People who want antibiotics for viral infections, or who refuse vaccines (covid and other) claiming they're "untested" or have "side effects" while promoting alternatives that both failed testing and have known side effects. I've seen people speak as if government taxation only exists because the guy in charge of taxes is, personally, greedy. I've heard anecdotes of people saying that you can get people to follow the rules by saying "first rule is to always follow the rules" and directly seen people talk as if banning something is sufficient to make it stop.
The idea that it's even possible to do make a model like this from the user data, is probably mind-blowing to a lot of people.
The naïve assumption most people seem to have is that computers do only what they, personally as end-users, tell them to do, and that they're as slow as the ad-riddled web front-end with needlessly slow transition animations placed there to keep user engagement high — rather than the truth, that software primarily does what the operator of the service wants it to do, and that it's absolutely possible for a home PC[0] to hold and query a database of all 8 billion people on the planet and the two trillion or so different personal relationships between them.
When GenAI images were new, some of the artists communities said "That content generated can reference hundreds, even thousands of pieces of work from other artists to create derivative images"[1], rather than millions of images, because the scale of computer performance is far beyond the comprehension of the average person. The fact that the average single image contributes so little to any given model that it can't even represent its own filename, even moreso.
And so it is with stuff like this: what can be done, cannot be comprehended by the people who, theoretically, gave consent that their data be used in that way.
[0] Of course, these days most people don't have home PCs; phone, perhaps a tablet, they may have a small low performance media server if they're fancy, but what we here would think of as a PC is to all that as a Ferrari etc. is to a Honda Civic.
[0] https://www.furaffinity.net/journal/10321622