Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts about Anthropic's new cryptanalysis results

blog.cryptographyengineering.com

Post reply on HN
Print view View original

101–110 of 124 posts

Re: Some thoughts about Anthropic's new cryptanalysis results

#101

Post by brookst »

inigyou wrote: post #51
ameliaquining wrote: post #31

Earlier quoted context omitted.

Shouldn't that imply that agnosticism is the proper view, rather than asserting that something is impossible on a next-token-predictor architecture? (Note: I don't actually think the consciousness question is the most important one in the near term. Where I think this line of reasoning gets really dangerous is when people use it to assert that LLMs can't or won't engage in certain behaviors no matter much they advanc…

If you believe that matrix multiplication with random sampling is conscious, then you probably believe everything is conscious, like rocks. Most people would expect that matrix multiplication is not conscious, and autocomplete is not conscious either. We can't prove matrix multiplication isn't conscious, but it doesn't seem likely unless everything is conscious.

Seems like a forest / trees error.

“If LLMs are conscious, it means matrix multiplication is conscious” == “If humans are conscious it means cells are conscious”.

It’s possible for complex systems to have emergent properties not exhibited by any individual component of the system.

Re: Some thoughts about Anthropic's new cryptanalysis results

#102

Post by ben_w »

sph87 wrote: post #65
simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

> AGI is already here I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting. Take for example Humanity's Last Exam. It is so broad and complex that while an individual in a specific field might be able to answer their specific area of questions, they certainly would not be able to achieve >50% on the total questi…

> I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting.

I'd say LLMs have shown us the opposite problem: there were many different definitions whose differences we'd previously been able to ignore. We don't all agree even on a single letter of "A", "G", and "I".

And this is why it looks like a moving goalpost.

Re: Some thoughts about Anthropic's new cryptanalysis results

#103

Post by ben_w »

mirekrusin wrote: post #78
kijin wrote: post #74

Earlier quoted context omitted.

We'll probably be able to tell if and when Yudkowsky's AGI arrives. Once AI can improve itself, frontier labs will no longer need human developers. So we might see a massive layoff of top talent and a dramatic increase in product quality at the same time. This usually doesn't happen in human businesses. It is also very much against the interest of anyone who is already at the top of the pay table at those labs.

Layoffs would be a poor indicator that recursive self-improvement had occurred as you need people with knowledge around domain/layer the work is done on. Also, if the lab truly has a self-improving superintelligence, the cost of retaining staff at any level would be a rounding error relative to its operating costs and the value the system creates. There would be little economic pressure to fire them immediately, espe…

> Layoffs would be a poor indicator that recursive self-improvement had occurred as you need people with knowledge around domain/layer the work is done on.

This is why I keep saying we can't all agree even on a single letter of "A", "G", and "I".

Before ChatGPT, I would have said "obviously a generally intelligent system can do all the things". While LLMs are much more general than AI before them, the quality of their performance in all the things is distributed in a very un-human-like way.

Some fast-moving optimiser can be a threat well before it stops needing any humans for part of their labour. Cancer and viruses are examples of this: they're the same category of thing as a paperclip optimiser, but for biology instead of manufacturing office supplies.

But some others will argue LLM-spikey isn't "AGI", they'll demand something which reaches the performance of the best human (or the mean human, or the mean domain expert, because we can't agree on "I"), and a standard of "≥ best human" would mean that no, you don't need "people with knowledge around domain/layer the work is done on".

Re: Some thoughts about Anthropic's new cryptanalysis results

#104

Post by arw0n »

sph87 wrote: post #65
simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

> AGI is already here I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting. Take for example Humanity's Last Exam. It is so broad and complex that while an individual in a specific field might be able to answer their specific area of questions, they certainly would not be able to achieve >50% on the total questi…

> I am not arguing that rote memorization is intelligence, or that we have achieved it, but does anyone know what AGI actually.. is?

I would argue that a core part of intelligence is being able to handle uncertainty. That + planning probably explains most of the evolutionary pressure for making our brains bigger. But if this is important to intelligence, chess bots in the 80s were more intelligent than their later counter-parts, which could simply remove uncertainty through rote memorization. Maybe the All-Knowing is a compete dud, no reasoning capabilities at all, just an extremely efficient, infinite lookup table of all facts.

Terms like intelligence and consciousness often just seem overloaded with meaning, and when discussing things concretely, we quickly switch to more specific terms like reasoning.

Re: Some thoughts about Anthropic's new cryptanalysis results

#105

Post by ruszki »

IanCal wrote: post #82
didibus wrote: post #64

Earlier quoted context omitted.

> the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different I'm not understanding, can you explain this more? How does it become more than a next token predictor? Isn't the post-training simply altering the sampled distribution? And isn't that distribution naturally occurring? It's the distribution of "useful" next token?

IMO that doesn’t sound so much like prediction any more. It’d be prediction if it’s “predict what would come next in this text sampled from distribution X”. But what’s it predicting if we’re looking for new useful outputs? It’s finding a distribution that’s useful, and generating tokens, but it’s not predicting what comes next in a known sequence.

Just because you change loss function or dataset, it doesn’t become not a prediction. Just a prediction of something else.

Re: Some thoughts about Anthropic's new cryptanalysis results

#106

Post by chasd00 »

jiggawatts wrote: post #54
simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

My mental model is this: There is a vast ocean of human knowledge, far beyond the capacity of any human brain, even within specialised fields. Books helped "plug the gaps" in our knowledge, increasing the scope that a single human mind can encompass. Web search engines did the same thing, but more and faster. LLMs are like search engines on steroids, essentially a research librarian that operates at 1,000x human spee…

> LLMs are like search engines on steroids

i like the analogy of a lossy compression algorithm. The LLM compresses all of the data it was trained on to answer the question it was asked.

Re: Some thoughts about Anthropic's new cryptanalysis results

#107

Post by ricksunny »

gwd wrote: post #98
simonw wrote: post #4

> both outputs of Claude Mythos, their (still) unreleased advanced model That sentence gives the impression that Mythos might be released in the future. That's clearly not going to happen - it's already "released" in as much as selected, trusted partners can access it, and the rest of us get it in the form of Fable - which is Mythos but with filters that downgrade you if you try to use it for anything even remotely r…

Got downgraded from Fable to Opus after asking about the Great Oxidation Event [1], presumably because it started thinking about cyanobacteria, which made the monitor afraid I was trying to make a biological weapon or something. [1] https://en.wikipedia.org/wiki/Great_Oxidation_Event

Were you trying to overoxidize all extant species?

Re: Some thoughts about Anthropic's new cryptanalysis results

#108

Post by Roark66 »

jiggawatts wrote: post #54
simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

My mental model is this: There is a vast ocean of human knowledge, far beyond the capacity of any human brain, even within specialised fields. Books helped "plug the gaps" in our knowledge, increasing the scope that a single human mind can encompass. Web search engines did the same thing, but more and faster. LLMs are like search engines on steroids, essentially a research librarian that operates at 1,000x human spee…

That is a good analogy. But the limitations of LLMs seem to be where they lack some information they tend to hallucinate/invent.

The rare benchmarks that measure "knowing what the model knows it doesn't know" show us there are only a couple models like Opus that are good in that field.

I think this is something that receives not enough attention from researchers.

Re: Some thoughts about Anthropic's new cryptanalysis results

#109

Post by dinfinity »

brookst wrote: post #97
kijin wrote: post #74

Earlier quoted context omitted.

We'll probably be able to tell if and when Yudkowsky's AGI arrives. Once AI can improve itself, frontier labs will no longer need human developers. So we might see a massive layoff of top talent and a dramatic increase in product quality at the same time. This usually doesn't happen in human businesses. It is also very much against the interest of anyone who is already at the top of the pay table at those labs.

AI is already improving itself. Most / all of the coding harnesses are AI-written. And yet… there are still people telling AI how to improve itself. IMO there will always be a level of abstraction at which AI needs guidance. Perhaps ASI means it decides everything on its own, but I don’t think so. Genius humans often excel at the how but not the why, or even the what. So far there’s no indication that AI is different…

> AI is already improving itself. Most / all of the coding harnesses are AI-written.

AI has not improved the network topology much yet. The next (and possibly 'last') big thing is enabling AI to come up with something as impactful as the transformer architecture.

Re: Some thoughts about Anthropic's new cryptanalysis results

#110

Post by simonw »

dinfinity wrote: post #109
brookst wrote: post #97

Earlier quoted context omitted.

AI is already improving itself. Most / all of the coding harnesses are AI-written. And yet… there are still people telling AI how to improve itself. IMO there will always be a level of abstraction at which AI needs guidance. Perhaps ASI means it decides everything on its own, but I don’t think so. Genius humans often excel at the how but not the why, or even the what. So far there’s no indication that AI is different…

> AI is already improving itself. Most / all of the coding harnesses are AI-written. AI has not improved the network topology much yet. The next (and possibly 'last') big thing is enabling AI to come up with something as impactful as the transformer architecture.

It's not exactly discovering the transformer, but GPT-5.6 Sol apparently just found optimizations that reduced OpenAI's cost of serving it by 20%: https://twitter.com/reach_vb/status/2082581596608376980

I'm guessing that's hundreds of millions and maybe even billions of dollars per month in savings.

Post reply on HN