11–20 of 100 posts
Excellent deep dive (and surprisingly not self-referential). I read the dead tree version ( https://amzn.to/4cs5EDA ) when it first came out.
You're right! I'm used to his articles being full of self promotion and Wolfram-talk, but this one is not.
The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…
My guess is that this is a post training (rlhf) artifact on world model prompts. There were likely many “logical inconsistency” prompts which humans coerced to the above response.
The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…
The problem with many early LLMs is that they could be persuaded into believing something if you stated that it’s true.
Maybe the default prompt in GPT-4o includes something to the effect of “You are a helpful, critical-thinking chatbot”…
The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…
That is one version of implication. The implication could also be that a time component isn't stated explicitly. He drove her to university. Time passes. Now it's "afterwards" and Sue is done with her classes and they go back home to drink tea. For all we know Joe had a cup of coffee while she was writing an exam but because they live so far from university it made no sense to go home and come back to come pick her u…
The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…
Another interesting bit of experiment people did when the GPT-4 class models launched were to test out spatial awareness. For eg, you could describe with words a construction made of blocks, spheres and so on and then ask questions about the stability of the structure. The newer models more often than not got it right.
Of course, that hasn't stopped people from making tired old claims of "stochastic parrot" and so forth but I think if your model can solve such reasoning problem (spatial or otherwise) then there's something really interesting to discover. I'm glad that folks at Anthropic are really trying to create a better understanding of these models (https://www.anthropic.com/research/mapping-mind-language-mod...)
Discussion then: https://news.ycombinator.com/item?id=34796611
The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…
This is an interesting thought exercise! The theory goes - that if the model "understands" this scenario, then internally, it has has created something that models the real world. Another interesting bit of experiment people did when the GPT-4 class models launched were to test out spatial awareness. For eg, you could describe with words a construction made of blocks, spheres and so on and then ask questions about th…
But this is merely a definition of what it means to "understand" something.
For example just tabulating many input/output combinations would not follow this definition.
Earlier quoted context omitted.
My guess is that this is a post training (rlhf) artifact on world model prompts. There were likely many “logical inconsistency” prompts which humans coerced to the above response.
Is RLHF same as fine tuning?
In essence, one is not telling the model "This. This is what you should output next time." but rather "I liked this reply. Have a cookie." The behaviors that you can learn in RL are more subtle, but you get a lot less information per step. That's because, in a causal language modeling objective, when I tell you "For the prompt X, you should output exactly Y[0...m)", you get a gradient for P(Y[0] | X), another one for P(Y[1] | X Y[0..1)), another for P(Y[2] | X Y[0..2)), another for P(Y[3] | X Y[0..3)), and so on. It's a lot more of a step-by-step guidance, than it is a sentence-wise reward that you get in the RL framework. In RL, I'd give you a cookie for P(Y | X). What part of Y made me give you that cookie? Was there even such a part? Was it perhaps some internal representation that made everything in Y better? That's for the model to learn.