Some thoughts about Anthropic's new cryptanalysis results • Hacker News Board

Re: Some thoughts about Anthropic's new cryptanalysis results

#61

Post by willy_k » Thu, Jul 30, 2026, 1:39 AM UTC

inigyou wrote: post #51

ameliaquining wrote: post #31

Earlier quoted context omitted.

Shouldn't that imply that agnosticism is the proper view, rather than asserting that something is impossible on a next-token-predictor architecture? (Note: I don't actually think the consciousness question is the most important one in the near term. Where I think this line of reasoning gets really dangerous is when people use it to assert that LLMs can't or won't engage in certain behaviors no matter much they advanc…

If you believe that matrix multiplication with random sampling is conscious, then you probably believe everything is conscious, like rocks. Most people would expect that matrix multiplication is not conscious, and autocomplete is not conscious either. We can't prove matrix multiplication isn't conscious, but it doesn't seem likely unless everything is conscious.

There are some vague notions about consciousness emerging from complexity that some people may advance as a nuance to your point, although that just makes rocks “minimally conscious”, not necessarily unconscious. If you take the view that the entirety of our consciousness’s comes from purely classical interactions (electrical and chemical), then thats not a difficult conclusion to arrive at, but the notion that consciousness can be derived from deterministic computation does not pass the sniff test imo.

As a side note, that is why I find the idea that the brain is a quantum-classical hybrid computer appealing. And following the research developments is very interesting, to say the least.

Re: Some thoughts about Anthropic's new cryptanalysis results

#62

Post by CamperBob2 » Thu, Jul 30, 2026, 2:16 AM UTC

thesz wrote: post #35

wren6991 wrote: post #25

Earlier quoted context omitted.

The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an op…

> The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different. https://arxiv.org/abs/2504.13837 "Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models a…

VibeThinker 3B is highly problematic for the thesis expressed in that paper. There is no possible way it could derive its reasoning capabilities from a bag of 3B parameters alone.

Re: Some thoughts about Anthropic's new cryptanalysis results

#63

Post by CamperBob2 » Thu, Jul 30, 2026, 2:21 AM UTC

simonw wrote: post #22

ameliaquining wrote: post #18

Earlier quoted context omitted.

I'm curious, what are you hoping to convey by reminding people that LLMs are next-token predictors? They are, of course, but most people without an AI background won't fully understand what that means, so I assume you're using it at least partly as a proxy for something else.

I think understanding how this stuff works is really important. For technical people it gives them a useful starting point for understanding it all. For less technical people it's crucial to help them understand that it's not some weird new magical science-fiction AI - it's still computer programs that turn text into numbers and do stuff with the numbers and turn those back into text. It's harder to believe something…

It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.

The problem is, these models challenge our definition of "consciousness." Or at least they point out how hopelessly-inadequate our thinking on the subject is. Some people really, really don't like having their personal definition of consciousness challenged.

The correct response to "So what, it's just a next-token predictor" isn't a long dissertation on RLHF, training architectures, scaling laws and whatever, but rather to turn around and respond, "Sure, and how is that different from what we do?"

Re: Some thoughts about Anthropic's new cryptanalysis results

#64

Post by didibus » Thu, Jul 30, 2026, 2:36 AM UTC

wren6991 wrote: post #25

simonw wrote: post #16

Earlier quoted context omitted.

Some people use it to demystify, but a whole lot of people seem to be using it to dismiss the technology entirely. Personally I like to remind people that these things are next-token predictors, but then emphasize how truly astonishing the results we can get out of sufficiently advanced next-token predictors are.

The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an op…

> the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different

I'm not understanding, can you explain this more? How does it become more than a next token predictor? Isn't the post-training simply altering the sampled distribution? And isn't that distribution naturally occurring? It's the distribution of "useful" next token?

Re: Some thoughts about Anthropic's new cryptanalysis results

#65

Post by sph87 » Thu, Jul 30, 2026, 2:43 AM UTC

simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

> AGI is already here

I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting.

Take for example Humanity's Last Exam. It is so broad and complex that while an individual in a specific field might be able to answer their specific area of questions, they certainly would not be able to achieve >50% on the total question set.

There is this idea that AI has to be perfect to be intelligent - but we consider Humans intelligent and they are not even close. So is it the ability to generate novel ideas? Prove theorems? Pass tests?

I am not arguing that rote memorization is intelligence, or that we have achieved it, but does anyone know what AGI actually.. is?

Re: Some thoughts about Anthropic's new cryptanalysis results

#66

Post by ameliaquining » Thu, Jul 30, 2026, 3:01 AM UTC

notahacker wrote: post #53

ameliaquining wrote: post #37

Earlier quoted context omitted.

Sorry, what contexts are these?

I don't know about "most", but there are a lot of people treating it as somewhere between "magic oracle" and "new friend" Not to mention people more worried about whether the AI is motivated to hurt us than what human motivations can do with something that can autocomplete its way through every possible attack vector of cryptographic systems most of use would prefer remain secure. (tbf I think the "glorified autocomp…

Whether an AI's operators lose control of it is an outcomes question.

Re: Some thoughts about Anthropic's new cryptanalysis results

#67

Post by annzabelle » Thu, Jul 30, 2026, 3:29 AM UTC

sph87 wrote: post #65

simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

> AGI is already here I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting. Take for example Humanity's Last Exam. It is so broad and complex that while an individual in a specific field might be able to answer their specific area of questions, they certainly would not be able to achieve >50% on the total questi…

There's a specific singularity theory of AI that is very popular. Eliezer Yudkowsky helped popularize it among the Bay Area "rationalist" community, and it has this idea that AGI necessarily implies a self improving system that will quickly become a paperclip maximizer or other such dystopian or utopian world changing intelligence.

By that singularity definition, we're probably nowhere near AGI, but if we define it as something that is as good at text/information manipulation as the 50th percentile human? I think we're already there.

Re: Some thoughts about Anthropic's new cryptanalysis results

#68

Post by swordsith » Thu, Jul 30, 2026, 3:34 AM UTC

john_strinlai wrote: post #3

> They [anthropic] appear to have just told it to get some results and then strapped its nose to the grindstone until it found some. it is fun how well this works. i cant find the link immediately (will look and edit with it), but somewhere in the " hello there the jacobian conjecture is false thanx " thread, someone brought up a different conjecture breakthrough where the prompts were basically just repeated "no, ke…

Could send random characters along with 'keep going' and it would change nothing, the seed is whats causing deviation between these responses.

Re: Some thoughts about Anthropic's new cryptanalysis results

#69

Post by annzabelle » Thu, Jul 30, 2026, 3:41 AM UTC

john_strinlai wrote: post #15

adamzenith wrote: post #13

Earlier quoted context omitted.

In response to your edit, you should check out Terry tao's chat gpt logs about the recent Jacobian result. The models are smart enough to brute force some things, but can cut to the meat much faster with good prompting

i read his, too. his replies are indeed more directed, but also quite short, unstructured, and natural sounding. if i recall, maybe 1 or 2 of his prompts exceeded 50(ish) words. in my head, the comparison is the multi-paragraph prompts (borderline essays) i would read in various communities on reddit and similar forums, that people (often self-proclaimed "prompt engineers") said were "required" to get good output. or…

I didn't use generative AI much until a couple months ago. My last employer had a copilot license that I dabbled in in 2024, and wasn't impressed with, and then I spent the latter half of 2025 and early 2026 budget traveling/hiking a lot. I started at a new company in May, and I've been astonished at how lazy I can be at prompting and still get impressive results with 2026 Claude Code.

I can paste entire failure logs with the word "why" lowercase, no question mark, and get into a productive chat session where it significantly speeds up the bug trace. I will paste the text from a groomed ticket with no editing or additional instructions into the chat and then give it a bit of feedback on the plan for a couple iterations.

I feel so vindicated in never spending time learning prompting as a specific skill, it really just took a couple more years and the models are really easy to interact with with simple natural language.

Re: Some thoughts about Anthropic's new cryptanalysis results

#70

Post by didibus » Thu, Jul 30, 2026, 4:08 AM UTC

wren6991 wrote: post #32

simonw wrote: post #27

Earlier quoted context omitted.

Right, but it's still useful to think of these models in terms of next-tokens because it helps explain that they look at every token that came before and use that to put out the next one. You can get into RL as part of explaining why it's so unnervingly good at picking a next token.

That's true. The fact that an LLM is a pure function of (all previous tokens) -> (next token), with internal state like KV cache only existing for optimisation purposes, is pretty mind-blowing. I guess it was more the "predictor" part I had issue with. There's a tendency to reach for statistical or probabilistic terminology to describe things that aren't usefully understood in those terms. For example in the "Speed A…

> which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it.

Isn't G a learned probability?