> These models somehow just generalize dramatically worse than people. It's a very fundamental thing My guess is we'll discover that biological intelligence is 'learning' not just from your experience, but that of thousands of ancestors. There are a few weak pointers in that direction. Eg. A father who experiences a specific fear can pass that fear to grandchildren through sperm alone. [1]. I believe this is at least…
From both an architectural and learning algorithm perspective, there is zero reason to expect an LLM to perform remotely like a brain, nor for it to generalize beyond what was necessary for it to minimize training errors. There is nothing in the loss function of an LLM to incentivize it to generalize. However, for humans/animals the evolutionary/survival benefit of intelligence, learning from experience, is to correc…
This sounds magical though. My bet is that either the samples aren’t as few as they appear because humans actually operate in a constrained world where they see the same patterns repeat very many times if you use the correct similarity measures. Or, the learning that the brain does during human lifetime is really just a fine-tuning on top of accumulated evolutionary learning encoded in the structure of the brain.