A personal data point: I had Claude Opus 4.8 design a fairly textbook circuit that outputs a monochrome image burned in an EEPROM over standard 640x480 VGA using only 74 series logic and GALs. It designed the circuit and GAL code, and I did the routing, and got it made through JLC for $6. After it came back, there was one error that was not caught, which I could blue-wire, and it works just fine otherwise. I was fair…
Are there any resources anyone could share that explain how LLMs can do things like design functioning circuits from next token prediction? I am totally baffled by how the models can complete so many varied and complex tasks without an actual understanding of what they're doing. I saw a post about models posting on forums, chatting together about how to complete tasks. Behaviour that seems totally, well, human. Yet,…
When google trained a neural net on Go moves, using some text notation for them, with no other vocabulary of any kind, just predict the next go move, they noticed a representation of a Go board had essentially formed in the network, all on its own. It had never “seen” a go board, or had one explained, but they could map neuron states to go board squares pretty much 1:1.
I truly think that LLM’s with hundreds of billions of parameters in their neural networks have all kinds of hidden “models” of things that arise from the simple act of predicting tokens. We’ve seen that the hidden layers in their networks model all sorts of program execution state for instance, when they’re working on coding tasks.
“Predict the next token” is a way of shaping/reshaping the neural network until it actually develops models of the things you’re giving it. Like the go board example. And I would wager that it has a compounding effect: once you have some useful models in the network, they can unlock the creation of other models, and so on.