Earlier quoted context omitted.
Are there any resources anyone could share that explain how LLMs can do things like design functioning circuits from next token prediction? I am totally baffled by how the models can complete so many varied and complex tasks without an actual understanding of what they're doing. I saw a post about models posting on forums, chatting together about how to complete tasks. Behaviour that seems totally, well, human. Yet,…
My 2¢: When google trained a neural net on Go moves, using some text notation for them, with no other vocabulary of any kind, just predict the next go move, they noticed a representation of a Go board had essentially formed in the network, all on its own. It had never “seen” a go board, or had one explained, but they could map neuron states to go board squares pretty much 1:1. I truly think that LLM’s with hundreds o…
I assume that you are referring to AlphaGo or AlphaZero. In either case, this statement is not correct. Both algorithms most certainly know exactly what a go board looks like, and what the rules are.
In the case of AlphaZero, it initially did not know how to best play the game, or what strategy or tactics would work. But the connections between the neural network and the go board are hardcoded, by humans.