41–50 of 124 posts
Earlier quoted context omitted.
Depends. "Most" implies majority, and the majority of people are using these tools not for programming but in contexts where ontology is more relevant than capability (not that capability is irrelevant, but most people care, or are tricked into caring, far more about the former).
Sorry, what contexts are these?
Earlier quoted context omitted.
The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an op…
> The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different. https://arxiv.org/abs/2504.13837 "Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models a…
Let's stipulate that what pretraining does is train next token prediction over a gigantic corpus. You can then sample from this distribution repeatedly (cf the Large Language Monkeys paper) and count how often it passes some deterministic verifier.
What GRPO-style RLVR does is precisely this, but then reward the trajectories which passed the verifier. These distributions are _by construction_ within the accessible output space of the pretrained model; you're reweighting the distribution so that pass@k goes up, because that's (for applications like programming) very useful. RLVR is about making sampling more efficient; the only new information being added to the system is the presence of the verifier, and note that you only get a reward when the verifier passes, so there's essentially no mechanism for "teaching new facts" here.
Earlier quoted context omitted.
I don't think the next-token-predictor thing should increase anyone's confidence that LLMs aren't conscious or can't escape the control of their operators. A very closely analogous argument would "prove" that humans aren't conscious or can't do [insert task here] either. (No, I'm not saying that any of this is true of today's LLMs, I'm saying this particular argument doesn't work.) I recommend this explanation: https…
You can say that for any argument regarding consciousness, because we don’t have an actual, all encompassing definition of what consciousness is. In general I don’t think comparison with humans makes much sense, we should be able to discuss LLMs without always falling back to “but what about humans” (sorry for the caricature)
But LLMs are only operating on text and humans are only operating on
Earlier quoted context omitted.
The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an op…
> The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different. https://arxiv.org/abs/2504.13837 "Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models a…
> Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model
On the face of it this seems unsurprising given the policy gradient term directly minimises this difference.
I don't have a good feel for how the output of an RLVR-trained model concretely differs from the base model. My guess would be there are a fairly small number of "forks" where the training creates a token flip that sends the model down a more useful path.
The fact that the straight paths between the forks resemble the base model would again be unsurprising since (a) those are exactly the right context to continue to elicit more output that's relevant to solving the problem (so not penalised by RLVR), and (b) preservation drops naturally out of the policy gradient term you add to limit catastrophic forgetting in the base model.
Low perplexity could be explained by the relative sparsity of the forks in the output stream, and/or by forks already having high entropy in the base model. That also aligns with the pass-at-high-k: yes it's doing more exploration without training but it's a bit of a monkeys-on-typewriters situation.
Lack of novelty is readily explained by the fact that you need some nonzero pass rate in the base model to actually get some useful training signal from RLVR. That's a limitation of contemporary RLVR techniques, not a limitation on post-training in general.
I think there's room in that forks-and-straights characterisation for the RLVR'd model to be doing something that looks a lot like computation, while having low perplexity vs the base model. I don't see anything in my admittedly incredibly shallow skim of the paper that refutes that.
> They [anthropic] appear to have just told it to get some results and then strapped its nose to the grindstone until it found some. it is fun how well this works. i cant find the link immediately (will look and edit with it), but somewhere in the " hello there the jacobian conjecture is false thanx " thread, someone brought up a different conjecture breakthrough where the prompts were basically just repeated "no, ke…
What a weird species of halting problem…
> They [anthropic] appear to have just told it to get some results and then strapped its nose to the grindstone until it found some. it is fun how well this works. i cant find the link immediately (will look and edit with it), but somewhere in the " hello there the jacobian conjecture is false thanx " thread, someone brought up a different conjecture breakthrough where the prompts were basically just repeated "no, ke…
That second person stated that for many years they tried that particular graph problem on various AI models, starting with o1 and o3. Its quite likely they now found the counterexample with a more serious prompt, and then for virality re-tried a few times with meme-prompts like "you should do a breakthrough", knowing that the model is capable of solving this particular one. Worst case the meme-prompts don't work and…
> both outputs of Claude Mythos, their (still) unreleased advanced model That sentence gives the impression that Mythos might be released in the future. That's clearly not going to happen - it's already "released" in as much as selected, trusted partners can access it, and the rest of us get it in the form of Fable - which is Mythos but with filters that downgrade you if you try to use it for anything even remotely r…
Genuine question here, why would you ask Fable to explain the difference between tusks and teeth? That's a task that can probably be handled by Haiku.
Now obviously you can and should retort with hallucination and confabulation rates from external and Ant's own reports per model (pretty sure more advanced models are good at lying better, not less) instead of going with dumb "more expensive more accurate" mental model, but general principle stands for me still AFAIK.
it's not exactly rational I admit, but if I'm going to base my own work and reasoning from an LLM I'm going with the best available. This seems to trip up most normies because they are too lazy or too greedy to pay up for premium access and see for themselves why most of us are both awed and afraid. Generally, I'm too biased and too deep in ML/DL cargo cult (been in it since 2016) to know if the skepticism and disdain for such usage is warranted.
In general, I think the tools are broadly toxic in a Dune-sense of making me think less for myself, because just as any HN-poster knows coding and doing mundane low-level stuff is necessary the same way doing stretches is necessary before any workout. The process itself is what keeps your brain strong and its gradients from veering into overfitting. I'm not overly bullish on the whole reaching for the stars ending with these things. Paradoxically, you using them eventually hobbles both you and the model, because you become dumber and then you bottleneck their ability to self-direct (broadly true for next Mythos/GPT-7).
Sorry for a long rant, was just anticipating some things I'd have to say for myself.
Earlier quoted context omitted.
In response to your edit, you should check out Terry tao's chat gpt logs about the recent Jacobian result. The models are smart enough to brute force some things, but can cut to the meat much faster with good prompting
i read his, too. his replies are indeed more directed, but also quite short, unstructured, and natural sounding. if i recall, maybe 1 or 2 of his prompts exceeded 50(ish) words. in my head, the comparison is the multi-paragraph prompts (borderline essays) i would read in various communities on reddit and similar forums, that people (often self-proclaimed "prompt engineers") said were "required" to get good output. or…
Not sure when you started. However: I would never judge the necessity of details provided 2 or 3 years ago based on results that the current models give.
Earlier quoted context omitted.
Shouldn't that imply that agnosticism is the proper view, rather than asserting that something is impossible on a next-token-predictor architecture? (Note: I don't actually think the consciousness question is the most important one in the near term. Where I think this line of reasoning gets really dangerous is when people use it to assert that LLMs can't or won't engage in certain behaviors no matter much they advanc…
I think you can reliably assert that X != Y without having a complete definition of Y, as long as you can identify at least one property or condition that Y possesses which X violates. So for consciousness and LLMs it could be Qualia, lack of semantic understanding, lack of continuity in time, lack of a high degree of integrated causal feedback, etc. Or perhaps those are just features of human consciousness but not i…
But qualia are not directly measurable and the rest of the list only matters if those features are necessary for qualia, which we can't decide without such measurements or at least a strong theoretical model.
Earlier quoted context omitted.
I'm also getting irritated with the “glorified autocomplete” comments. Since nobody can post such comments and also use the tools I'm using, I'm wondering if the phenomenon is due to people only having experience with the free version of whatever it is they're trying to use?
The “glorified autocomplete” framing isn’t to take literally. It’s a way to remove the mystic and whole anthropomorphization of AI. It’s saying they aren’t sentient or entities we are interacting with, even if that’s how the output presents itself. Instead they are “just” stochastic models