Earlier quoted context omitted.
Just because you change loss function or dataset, it doesn’t become not a prediction. Just a prediction of something else.
What’s it a prediction of?
Prediction of the next token based on dataset and loss function.
Some thoughts about Anthropic's new cryptanalysis results
121–124 of 124 posts
Re: Some thoughts about Anthropic's new cryptanalysis results
#122Earlier quoted context omitted.
> The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different. https://arxiv.org/abs/2504.13837 "Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models a…
Ooh. This looks like an interesting paper and there were a couple of things in the intro that I found counter-intuitive. It'll take me a while to digest the whole thing. > Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model On the face of it this seems unsurprising given the policy gradient term directly minimises this difference. I don't have a…
Other post-train mechanisms have meaningful bandwidth for introducing new information to the model (on-policy distillation is more or less the other extreme). RLVR aligns the model to particular behaviors it could already (unreliably, intermittently) express, it doesn't introduce new behaviors; the behaviors were latent in the pretrain/midtrain.
Re: Some thoughts about Anthropic's new cryptanalysis results
#123Re: Some thoughts about Anthropic's new cryptanalysis results
#124This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…
There is grotesque hype for this new category of software. And the skittish AGI debate generates more noise than it's worth.
Do use the tool if it offers material/measurable improvement for your use-case, but budget carefully and keep your people. They probably do actually know what your material/measurable use-case really is.
If you only talk to yezbotz, you may zuck yourzelf into a corner and look like a zhmendrick.