21–23 of 23 posts
"DeepSeek-V3-Base already exhibit 'Aha moment'."
I tried to read the screenshot they present as evidence of this, and indeed it does say "Aha!". But both the preceding reasoning and the following conclusion look like gibberish to me. I'm not sure what we're supposed to conclude here and I gave up reading the article after this inauspicious start.
Overall the industry needs more review, less hype. I was shocked to find out SWE-verified [0] is all but verified. [0] benchmark used by all major vendors to "showcase" coding ability, turns out to be https://www.youtube.com/watch?v=QnOc_kKKuac
Failure modes are also interesting to show what is happening or not really happening. Like the test of asking GenAI to create clocks at specific times, or people drawing with the left hand. All you get are clocks, at 10 min past two, or people drawing with the right hand, since it's 99% of what is in the training data. Like Sabine says, if the LLM models, already read all the Math books in the world but are not yet a…
Having trained quite a few LLMs by now, especially around the “uplift” from a text completion model to an instruct one, I noticed that the instruction following capabilities tend not to be uniform across all the tasks the LLM is able to perform.
In other words, it doesn’t know what is the implication of your request (time on the clock) in terms of change in output (the svg drawing of the clock), and all the abstractions and steps in between, this is why changing the time in the prompt might yield little difference.
Instruct datasets are quite small in retrospect and hardly cover the range of tasks instruct LLMs are able to perform, and mostly rely on extrapolation (interpolation?) capabilities for the LLM from its small training set. And it is this generalization that is not evenly distributed.
Just my two cents from my modest observations, sprinkle [citation needed] everywhere.