Earlier quoted context omitted.
Another interesting facet is that, according to some schools of thought, compression and AI are equivalent problems: The better you understand something, the less you need to memorize to reproduce it. https://en.wikipedia.org/wiki/Hutter_Prize (and https://en.wikipedia.org/wiki/AIXI ) Of course, large language models are (by definition) currently going the other direction, but it remains to be seen whether that leads…
> Of course, large language models are (by definition) currently going the other direction ... How so? Aren't the networks' weights orders of magnitude smaller than the training data?
But they don't losslessly recreate the training data.