Facebook scraped every Australian adult user's public posts to train AI

OK, but I think there's a pretty big difference between "Next.js is too bloated, you should use HTMX" and "so and so group of people are all _____ and they should all be ________, and oh, your mom sucks".

I don't think – I hope, at least – no one is going to start a shooting war over their framework of choice. You can't say the same about much of the content circulating around social media.

(Edit: You added more to your post after I replied. To your point of "Which HN avoids by usually remaining focused on a fairly narrow set of subjects", that's not just my observation, that's the actual guidelines: https://news.ycombinator.com/newsguidelines.html. We self-select into a narrow slice of nerdtalk or we end up getting downvoted or banned from the site. To me that is the big difference between an interest-based forum that generally stays a functional monoculture vs a general social media site that brings diverse strangers together into shouting wars about whatever the controversy du jour is.)

> I'm not convinced that the philosophy of copyright is a net positive for society.

I'm ok with that. But the philosophy of copyrights is not under debate here. All that is being debated is if it should protect small people from big corporations too.

> You could make the same argument about paper. "That's just how photocopiers work! If you don't want your creations to be endlessly duplicated and sold, don't write them down!"

No, the argument would be about photocopies, not paper. "That's just how photocopiers work! Don't put something into a photocopier if you don't want photocopies of it." It isn't possible for anyone to access anything on the internet without making copies of that thing. Copies are literally how the internet works.

Shooting everyone who steps outside isn't how guns work either so that also fails as an analogy.

The internet was specifically designed for the global distribution of copies. If that isn't what you want, don't publish your works there.

> That something is technically possible doesn't make it morally right.

Morality is entirely different from how the internet works, but in practice, I don't see anything immoral about making a copy of something. Morality only becomes an issue when it comes to what someone does with that copy.

Yes; but in the case of a "common misconception" like this, there's always also a nontrivial minority who do know the "right answer"[1] — and so enough examples of that occur in the training data to enable the model to embed the knowledge-web of "right ideas" (as a niche activation), alongside the "wrong idea" (in its default-mode network).

The "initial fine-tuning of a 'raw' base model to produce a 'generalized pre-trained' base model" process, is commonly talked about in terms of "alignment" — making the model ethical, making it not swear at you, making it refuse to engage with certain content, etc. And really, that part is all optional, with there existing "non-aligned" or "orthogonalized" models that don't have these steps performed on them or have had them reversed, but which are still useful.

But a large part of this initial fine-tuning process, consists of debiasing the model's default activation, moving it away from making associations with "common misconceptions" and toward making associations with "right answers." And this process is crucial to a model being able to reason intelligently — as these common misconceptions aren't coherent in a chain of reasoning they appear in, and so lead to the chain of reasoning falling apart / being non-productive. This is, in large part, the "secret sauce" that makes a model of a given size "more intelligent" than another model of the same size.

Every base model that anyone actually cares about or uses — "aligned" or not — has had some process of de-biasing like this applied to it; or, at least, has short-cutted this process by training on a training dataset generated by or filtered by a model that has already had this de-biasing applied to it, such that the derived training dataset doesn't contain the "common misconceptions" in the first place.

And when OpenAI and Meta brag about using RLHF, a large part of what they mean, is crowdsourcing recognition of long-tail "common misconceptions" at scale, to allow a much more thorough version of this de-biasing process[2].

---

[1] Of course, if nobody in the training data ever demonstrates the "right answer" knowledge/associations, then the model will never learn that knowledge/associations. But then, given that these training datasets usually represent decent samples of the population, a "right answer" being missing entirely would likely mean that nobody on Earth knows the "right answer" — so we humans wouldn't be able to recognize the model was wrong. The stock market can be wrong too, for the same reason.

[2] Which, perhaps surprisingly, can add up to more than the sum of its parts. The more of this human-labelled "common misconception"-response RLHF data you have, the more you can derive patterns from this bias data. You can distil out negative examples that you can use to prompt a model to filter the training dataset; but more interestingly, you can distil positive examples of the sort of structured chains of reasoning that work best to inherently avoid triggering the bias. Where, if you can overlap the activation-space hyperspheres of many such inversion-of-bias examples, then you get, essentially, the hypersphere within the model's activation space that contains its instrumental rationality. You can then just bias the model toward living in that part of activation space as much as possible — and this shoots its apparent reasoning capacity way up.