Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts about Anthropic's new cryptanalysis results

blog.cryptographyengineering.com

Post reply on HN
Print view View original

31–40 of 124 posts

Re: Some thoughts about Anthropic's new cryptanalysis results

#31

Post by ameliaquining »

dgellow wrote: post #30
ameliaquining wrote: post #26

Earlier quoted context omitted.

I don't think the next-token-predictor thing should increase anyone's confidence that LLMs aren't conscious or can't escape the control of their operators. A very closely analogous argument would "prove" that humans aren't conscious or can't do [insert task here] either. (No, I'm not saying that any of this is true of today's LLMs, I'm saying this particular argument doesn't work.) I recommend this explanation: https…

You can say that for any argument regarding consciousness, because we don’t have an actual, all encompassing definition of what consciousness is. In general I don’t think comparison with humans makes much sense, we should be able to discuss LLMs without always falling back to “but what about humans” (sorry for the caricature)

Shouldn't that imply that agnosticism is the proper view, rather than asserting that something is impossible on a next-token-predictor architecture?

(Note: I don't actually think the consciousness question is the most important one in the near term. Where I think this line of reasoning gets really dangerous is when people use it to assert that LLMs can't or won't engage in certain behaviors no matter much they advance; this doesn't have anything to do with consciousness.)

Re: Some thoughts about Anthropic's new cryptanalysis results

#32

Post by wren6991 »

simonw wrote: post #27
wren6991 wrote: post #25

Earlier quoted context omitted.

The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an op…

Right, but it's still useful to think of these models in terms of next-tokens because it helps explain that they look at every token that came before and use that to put out the next one. You can get into RL as part of explaining why it's so unnervingly good at picking a next token.

That's true. The fact that an LLM is a pure function of (all previous tokens) -> (next token), with internal state like KV cache only existing for optimisation purposes, is pretty mind-blowing.

I guess it was more the "predictor" part I had issue with. There's a tendency to reach for statistical or probabilistic terminology to describe things that aren't usefully understood in those terms. For example in the "Speed Always Wins" LLM technical survey (https://arxiv.org/pdf/2508.09834):

> The gate is a crucial component to bring sparsity in MoE models. For a batch of input token representations X ∈ RT×D, the gate function G determines the probabilities of dispatching token xi to each expert e

...which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it.

Re: Some thoughts about Anthropic's new cryptanalysis results

#33

Post by ameliaquining »

argee wrote: post #28
dgellow wrote: post #14

Earlier quoted context omitted.

The “glorified autocomplete” framing isn’t to take literally. It’s a way to remove the mystic and whole anthropomorphization of AI. It’s saying they aren’t sentient or entities we are interacting with, even if that’s how the output presents itself. Instead they are “just” stochastic models

Exactly. They ARE "glorified autocomplete" in an ontological sense. That says nothing about capability or outcome. The people who come out swinging against that characterization usually ignore the whole ontological argument (which is...the entire point) and go after an outcome-based strawman.

Isn't the outcomes question the one that people actually care about in most contexts?

Re: Some thoughts about Anthropic's new cryptanalysis results

#34

Post by mostlylurks »

dboreham wrote: post #12
simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

I'm also getting irritated with the “glorified autocomplete” comments. Since nobody can post such comments and also use the tools I'm using, I'm wondering if the phenomenon is due to people only having experience with the free version of whatever it is they're trying to use?

The models themselves are indeed glorified autocomplete in terms of what they actually do (with things like agentic coding harnesses being required as a wrapper around them to make that internal autocomplete something more useful). Many people use this fact to critique LLMs, but many other common instances of people pointing out LLMs' apparent lack of intelligence actually come from people not understanding that the model is a glorified autocomplete underneath whatever interface people access them through, and the interface isn't providing the underlying model all the information they assume it would, making it seem less intelligent than it actually is.

Re: Some thoughts about Anthropic's new cryptanalysis results

#35

Post by thesz »

wren6991 wrote: post #25
simonw wrote: post #16

Earlier quoted context omitted.

Some people use it to demystify, but a whole lot of people seem to be using it to dismiss the technology entirely. Personally I like to remind people that these things are next-token predictors, but then emphasize how truly astonishing the results we can get out of sufficiently advanced next-token predictors are.

The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an op…

  > The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.
https://arxiv.org/abs/2504.13837

"Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. "

Re: Some thoughts about Anthropic's new cryptanalysis results

#36

Post by argee »

ameliaquining wrote: post #33
argee wrote: post #28

Earlier quoted context omitted.

Exactly. They ARE "glorified autocomplete" in an ontological sense. That says nothing about capability or outcome. The people who come out swinging against that characterization usually ignore the whole ontological argument (which is...the entire point) and go after an outcome-based strawman.

Isn't the outcomes question the one that people actually care about in most contexts?

Depends. "Most" implies majority, and the majority of people are using these tools not for programming but in contexts where ontology is more relevant than capability (not that capability is irrelevant, but most people care, or are tricked into caring, far more about the former).

Re: Some thoughts about Anthropic's new cryptanalysis results

#37

Post by ameliaquining »

argee wrote: post #36
ameliaquining wrote: post #33

Earlier quoted context omitted.

Isn't the outcomes question the one that people actually care about in most contexts?

Depends. "Most" implies majority, and the majority of people are using these tools not for programming but in contexts where ontology is more relevant than capability (not that capability is irrelevant, but most people care, or are tricked into caring, far more about the former).

Sorry, what contexts are these?

Re: Some thoughts about Anthropic's new cryptanalysis results

#38

Post by slopinthebag »

simonw wrote: post #16
dgellow wrote: post #14

Earlier quoted context omitted.

The “glorified autocomplete” framing isn’t to take literally. It’s a way to remove the mystic and whole anthropomorphization of AI. It’s saying they aren’t sentient or entities we are interacting with, even if that’s how the output presents itself. Instead they are “just” stochastic models

Some people use it to demystify, but a whole lot of people seem to be using it to dismiss the technology entirely. Personally I like to remind people that these things are next-token predictors, but then emphasize how truly astonishing the results we can get out of sufficiently advanced next-token predictors are.

I think this is smart, it seems to me that the best way to use these models its to approach them as a next-token predictor instead of an intelligent entity. That's how I've gotten the best results from them and allows me to avoid some of the pitfalls people fall into by anthropromising them.

Re: Some thoughts about Anthropic's new cryptanalysis results

#39

Post by slopinthebag »

ameliaquining wrote: post #31
dgellow wrote: post #30

Earlier quoted context omitted.

You can say that for any argument regarding consciousness, because we don’t have an actual, all encompassing definition of what consciousness is. In general I don’t think comparison with humans makes much sense, we should be able to discuss LLMs without always falling back to “but what about humans” (sorry for the caricature)

Shouldn't that imply that agnosticism is the proper view, rather than asserting that something is impossible on a next-token-predictor architecture? (Note: I don't actually think the consciousness question is the most important one in the near term. Where I think this line of reasoning gets really dangerous is when people use it to assert that LLMs can't or won't engage in certain behaviors no matter much they advanc…

I think you can reliably assert that X != Y without having a complete definition of Y, as long as you can identify at least one property or condition that Y possesses which X violates.

So for consciousness and LLMs it could be Qualia, lack of semantic understanding, lack of continuity in time, lack of a high degree of integrated causal feedback, etc.

Or perhaps those are just features of human consciousness but not integral to consciousness as a whole. To me this then implies panpsychism to some degree, which I'm alright with too.

Re: Some thoughts about Anthropic's new cryptanalysis results

#40

Post by cyanydeez »

dgellow wrote: post #14
dboreham wrote: post #12

Earlier quoted context omitted.

I'm also getting irritated with the “glorified autocomplete” comments. Since nobody can post such comments and also use the tools I'm using, I'm wondering if the phenomenon is due to people only having experience with the free version of whatever it is they're trying to use?

The “glorified autocomplete” framing isn’t to take literally. It’s a way to remove the mystic and whole anthropomorphization of AI. It’s saying they aren’t sentient or entities we are interacting with, even if that’s how the output presents itself. Instead they are “just” stochastic models

I like glorified copy/paste frankensteined with find/replace.
Post reply on HN