> A LLM (...) merely produces syntactically likely combinations of the training data While doesn't matter much in the rest of the policy, this is a common misconception among AI skeptics. It is not the case for a long time (since RL is used heavily in the training) and a LLM may go beyond its training data.
The natural next argument that I see a lot is "well it's still all probabilistic", which is technically true. However, don't the atoms that make up the cells that make up a human move around and interact based on probabilities? Saying a LLM is all just based on probabilities is pretty meaningless, even if it is true, since everything else is just based on probabilities too. What matters is the massive amount of machi…
But there are more probabilistic elements about LLMs. Inference is by default deterministic for any given input and weights, however it is made stochastic algorithmically, when the model chooses the second or third likeliest outcome with some probability. Training is stochastic, the adjustment of the weights is probabilistic, etc. etc.
You seem to make the fact that our brains are also probabilistic as some sort of a gotcha. However that has never been in dispute. Neurons fire with certain probability, this has been known since the advent of neuroscience. However this is why computers are preferred for certain tasks over human brains. When you run a program you can be certain it behaves in a certain way given a particular input and parameters.