Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts about Anthropic's new cryptanalysis results

blog.cryptographyengineering.com

Post reply on HN
Print view View original

111–120 of 124 posts

Re: Some thoughts about Anthropic's new cryptanalysis results

#111

Post by KPGv2 »

notathrowaway51 wrote: post #75
KPGv2 wrote: post #60

Earlier quoted context omitted.

> At the risk of seeming X, statement that overwhelmingly demonstrates X-ness. It's exhausting to even consider where to begin addressing the assertion that good leadership is just predicting the next word to say. Especially considering the corpus available to most great leaders in history was extremely small. To think Hannibal's military campaigns were just because he'd read like ten books in his life and could accu…

"Competent strategy" is just as good a target for a word predictor to optimize as "effective rhetoric" is.

I suggest you re-read what I wrote. In particular the part where I said there was no large corpus to perform this prediction back then.

After you edit, I'll address your point about how military strategy is nothing but skill with words.

Re: Some thoughts about Anthropic's new cryptanalysis results

#112

Post by Forgeties79 »

KPGv2 wrote: post #111
notathrowaway51 wrote: post #75

Earlier quoted context omitted.

"Competent strategy" is just as good a target for a word predictor to optimize as "effective rhetoric" is.

I suggest you re-read what I wrote. In particular the part where I said there was no large corpus to perform this prediction back then. After you edit, I'll address your point about how military strategy is nothing but skill with words.

> After you edit, I'll address your point

This has to be the most insufferable thing I’ve read/heard in weeks. Are you being serious right now?

Re: Some thoughts about Anthropic's new cryptanalysis results

#113

Post by brookst »

dinfinity wrote: post #109
brookst wrote: post #97

Earlier quoted context omitted.

AI is already improving itself. Most / all of the coding harnesses are AI-written. And yet… there are still people telling AI how to improve itself. IMO there will always be a level of abstraction at which AI needs guidance. Perhaps ASI means it decides everything on its own, but I don’t think so. Genius humans often excel at the how but not the why, or even the what. So far there’s no indication that AI is different…

> AI is already improving itself. Most / all of the coding harnesses are AI-written. AI has not improved the network topology much yet. The next (and possibly 'last') big thing is enabling AI to come up with something as impactful as the transformer architecture.

It’s a fair point, but I’d argue “AI” is the whole stack, not just model weights. And AI is absolutely improving the whole stack in ways that make it smarter (from a consumer’s perspective).

Re: Some thoughts about Anthropic's new cryptanalysis results

#114

Post by [deleted] »

red75prime wrote: post #73
argee wrote: post #28

Earlier quoted context omitted.

Exactly. They ARE "glorified autocomplete" in an ontological sense. That says nothing about capability or outcome. The people who come out swinging against that characterization usually ignore the whole ontological argument (which is...the entire point) and go after an outcome-based strawman.

Capability or outcome...or internal structure. What does "glorified autocomplete" say in an ontological sense exactly? Nah. It's just a lazy dismissal. BTW, autoregressive pretraining (autocomplete) is a part of training.

[deleted]

Re: Some thoughts about Anthropic's new cryptanalysis results

#115

Post by IanCal »

ruszki wrote: post #105
IanCal wrote: post #82

Earlier quoted context omitted.

IMO that doesn’t sound so much like prediction any more. It’d be prediction if it’s “predict what would come next in this text sampled from distribution X”. But what’s it predicting if we’re looking for new useful outputs? It’s finding a distribution that’s useful, and generating tokens, but it’s not predicting what comes next in a known sequence.

Just because you change loss function or dataset, it doesn’t become not a prediction. Just a prediction of something else.

What’s it a prediction of?

Re: Some thoughts about Anthropic's new cryptanalysis results

#116

Post by xnx »

sph87 wrote: post #65
simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

> AGI is already here I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting. Take for example Humanity's Last Exam. It is so broad and complex that while an individual in a specific field might be able to answer their specific area of questions, they certainly would not be able to achieve >50% on the total questi…

ARC-AGI has thought a fair deal about this and has a good definition here: https://arcprize.org/arc-agi#defining-agi:~:text=AGI%20is%20....

"The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty."

Re: Some thoughts about Anthropic's new cryptanalysis results

#117

Post by tptacek »

irishcoffee wrote: post #94
tptacek wrote: post #84

Earlier quoted context omitted.

Subtextually, when people discuss AGI as a threshold, they're talking about superintelligence, which I'd roughly define as reasoning (not just applying algorithmic search) at the speed with which computers do lookups and computations, with large-scale data set contexts. Right now, models are doing well-defined knowledge work tasks, applying relatively well-worn patterns (but with the thoroughness of a computer, which…

Super intelligence is reasoning quickly? That’s what we’ve reduced it to? Don’t tell Mozart.

Superintelligence != Super intelligence.

Re: Some thoughts about Anthropic's new cryptanalysis results

#118

Post by dinfinity »

brookst wrote: post #113
dinfinity wrote: post #109

Earlier quoted context omitted.

> AI is already improving itself. Most / all of the coding harnesses are AI-written. AI has not improved the network topology much yet. The next (and possibly 'last') big thing is enabling AI to come up with something as impactful as the transformer architecture.

It’s a fair point, but I’d argue “AI” is the whole stack, not just model weights. And AI is absolutely improving the whole stack in ways that make it smarter (from a consumer’s perspective).

True, but that is clearly not where the biggest potential gains lie. The number of different topologies we've tried for ANNs is miniscule compared to the number of biological neural networks evolution has tried. The latter are also far more intricately organized.

Honestly, when it comes to fundamental ANN topology improvement we've only just gotten started.

Re: Some thoughts about Anthropic's new cryptanalysis results

#119

Post by Groxx »

red75prime wrote: post #88
Groxx wrote: post #58

Earlier quoted context omitted.

I generally like it still. It describes their failure modes pretty well, and in a way that most people already recognize. They're incomparably more complex, of course, but they are not intelligent and they are very much repeating what they've seen without any capability for factual accuracy. Practically every other attempt at describing them leans too technical and unfamiliar (stochastic parrot) or too anthropomorphi…

"Repeating what they've seen" could be, depending on their generalization abilities, something like low-n-gram Markov chain, repeating of surface-level speech patterns, repeating reasoning patterns, repeating mind-theory-level patterns, repeating self-correction patterns and so on.

And they fall somewhere between "surface-level speech patterns" and "repeating reasoning patterns" imo, yeah. They do indeed learn higher-order patterns than a Markov chain... but they'll blatantly misapply reasoning patterns and self-contradict confidently, just like you see with human bullshitters who don't understand one bit of what they're doing.

They sound fairly human, until you notice the patterns. They sound like they're thinking, until you pay attention.

Re: Some thoughts about Anthropic's new cryptanalysis results

#120

Post by didibus »

IanCal wrote: post #82
didibus wrote: post #64

Earlier quoted context omitted.

> the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different I'm not understanding, can you explain this more? How does it become more than a next token predictor? Isn't the post-training simply altering the sampled distribution? And isn't that distribution naturally occurring? It's the distribution of "useful" next token?

IMO that doesn’t sound so much like prediction any more. It’d be prediction if it’s “predict what would come next in this text sampled from distribution X”. But what’s it predicting if we’re looking for new useful outputs? It’s finding a distribution that’s useful, and generating tokens, but it’s not predicting what comes next in a known sequence.

> predict what would come next in this text sampled from distribution X

It tries to learn the distribution of "useful" results either through verified rewards or human feedback. Then it encodes that in the network. When you run inference later, it samples or selects from that distribution.

Maybe it's a matter of interpretation. It's not predicting the next token based purely on the training corpus's distribution anymore, the RL process fine tunes that distribution so it predicts the next token that is closer to what was rewarded during RL. But as I see it, it's still predicting the next token, just from a reenforcement learned distribution instead of one found in a corpus of data.

Post reply on HN