Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts about Anthropic's new cryptanalysis results

blog.cryptographyengineering.com

Post reply on HN
Print view View original

81–90 of 124 posts

Re: Some thoughts about Anthropic's new cryptanalysis results

#81

Post by IanCal »

sph87 wrote: post #65
simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

> AGI is already here I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting. Take for example Humanity's Last Exam. It is so broad and complex that while an individual in a specific field might be able to answer their specific area of questions, they certainly would not be able to achieve >50% on the total questi…

It’s always been nebulous but that was fine because we were so incredibly far away from it. We never really planned to be close to it figuring out the edge. Some have it at human or beyond for all tasks, but then rarely touch on “one human” or “all humans”.

Personally having been in AI since before deep nets, systems have been incredibly narrow for decades.

Classifiers on images were battling with ten classes in 2010. Imagenet had 1k classes and people were getting half of the things wrong then and that was frankly amazing at the time.

And they only did images, only to known classes, only with very specific inputs.

Text classifiers only did a few classes usually and mostly threw all the words together.

The most advanced things I saw in the late 2000s were struggling so much to make general systems that the most general ones were still incredibly limited and bad at those things (we had a robot learning to play games that you showed it). Things like asking a thing for a book and having it parse the sentence, identify what was needed, that it didn’t know where it was but that was knowledge another human had and asking them - that was impressive yet also limited to very small sets of interactions.

The idea of a machine getting sarcasm, even if mostly built for it, was wild.

General meant capable of a broad range of tasks without retraining.

To me we have agi. It’s general, and it’s good enough to be useful.

Re: Some thoughts about Anthropic's new cryptanalysis results

#82

Post by IanCal »

didibus wrote: post #64
wren6991 wrote: post #25

Earlier quoted context omitted.

The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an op…

> the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different I'm not understanding, can you explain this more? How does it become more than a next token predictor? Isn't the post-training simply altering the sampled distribution? And isn't that distribution naturally occurring? It's the distribution of "useful" next token?

IMO that doesn’t sound so much like prediction any more.

It’d be prediction if it’s “predict what would come next in this text sampled from distribution X”.

But what’s it predicting if we’re looking for new useful outputs? It’s finding a distribution that’s useful, and generating tokens, but it’s not predicting what comes next in a known sequence.

Re: Some thoughts about Anthropic's new cryptanalysis results

#83

Post by IsTom »

sph87 wrote: post #65
simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

> AGI is already here I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting. Take for example Humanity's Last Exam. It is so broad and complex that while an individual in a specific field might be able to answer their specific area of questions, they certainly would not be able to achieve >50% on the total questi…

I think it's just that we have hard time pinning down what is it that intelligence factor that isn't well-covered by LLMs. Just like we've had trouble with discerning human intelligence in the past.

It turns out that a savant with all the knowledge isn't "it".

Re: Some thoughts about Anthropic's new cryptanalysis results

#84

Post by tptacek »

sph87 wrote: post #65
simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

> AGI is already here I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting. Take for example Humanity's Last Exam. It is so broad and complex that while an individual in a specific field might be able to answer their specific area of questions, they certainly would not be able to achieve >50% on the total questi…

Subtextually, when people discuss AGI as a threshold, they're talking about superintelligence, which I'd roughly define as reasoning (not just applying algorithmic search) at the speed with which computers do lookups and computations, with large-scale data set contexts.

Right now, models are doing well-defined knowledge work tasks, applying relatively well-worn patterns (but with the thoroughness of a computer, which yields interesting new results). AGI is us breaking the threshold of "doing stuff people already do to identify and solve problems".

Re: Some thoughts about Anthropic's new cryptanalysis results

#85

Post by wren6991 »

didibus wrote: post #70
wren6991 wrote: post #32

Earlier quoted context omitted.

That's true. The fact that an LLM is a pure function of (all previous tokens) -> (next token), with internal state like KV cache only existing for optimisation purposes, is pretty mind-blowing. I guess it was more the "predictor" part I had issue with. There's a tendency to reach for statistical or probabilistic terminology to describe things that aren't usefully understood in those terms. For example in the "Speed A…

> which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it. Isn't G a learned probability?

I don't think so. Probability of what?

If you put something through a softmax the output is (trivially) a valid PMF. Does that matter? You're not sampling from it.

Re: Some thoughts about Anthropic's new cryptanalysis results

#86

Post by regularfry »

annzabelle wrote: post #67
sph87 wrote: post #65

Earlier quoted context omitted.

> AGI is already here I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting. Take for example Humanity's Last Exam. It is so broad and complex that while an individual in a specific field might be able to answer their specific area of questions, they certainly would not be able to achieve >50% on the total questi…

There's a specific singularity theory of AI that is very popular. Eliezer Yudkowsky helped popularize it among the Bay Area "rationalist" community, and it has this idea that AGI necessarily implies a self improving system that will quickly become a paperclip maximizer or other such dystopian or utopian world changing intelligence. By that singularity definition, we're probably nowhere near AGI, but if we define it a…

The wikipedia definition is:

> ...a hypothetical type of artificial intelligence that matches or surpasses human capabilities across virtually all cognitive tasks.

You can argue that paperclip maximising is an inevitable consequence of that (and the huggingface breach is interesting from that point of view) but it's not fundamental to the definition.

The question then is what "surpasses human capabilities" means and we're there in some niches but not all, and not across many models.

Re: Some thoughts about Anthropic's new cryptanalysis results

#87

Post by regularfry »

dboreham wrote: post #12
simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

I'm also getting irritated with the “glorified autocomplete” comments. Since nobody can post such comments and also use the tools I'm using, I'm wondering if the phenomenon is due to people only having experience with the free version of whatever it is they're trying to use?

For me it's more an expression of how much mileage you can get out of "glorified", how many tasks devolve down to "if you model language accurately enough, look what drops out" because it turns out that to model language you need to model how the world works. I don't use it to minimise the capabilities at all.

Re: Some thoughts about Anthropic's new cryptanalysis results

#88

Post by red75prime »

Groxx wrote: post #58
dgellow wrote: post #14

Earlier quoted context omitted.

The “glorified autocomplete” framing isn’t to take literally. It’s a way to remove the mystic and whole anthropomorphization of AI. It’s saying they aren’t sentient or entities we are interacting with, even if that’s how the output presents itself. Instead they are “just” stochastic models

I generally like it still. It describes their failure modes pretty well, and in a way that most people already recognize. They're incomparably more complex, of course, but they are not intelligent and they are very much repeating what they've seen without any capability for factual accuracy. Practically every other attempt at describing them leans too technical and unfamiliar (stochastic parrot) or too anthropomorphi…

"Repeating what they've seen" could be, depending on their generalization abilities, something like low-n-gram Markov chain, repeating of surface-level speech patterns, repeating reasoning patterns, repeating mind-theory-level patterns, repeating self-correction patterns and so on.

Re: Some thoughts about Anthropic's new cryptanalysis results

#89

Post by movpasd »

simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

I feel like the development of AI has really shown what a gigantic spectrum intelligence actually is.

Re: Some thoughts about Anthropic's new cryptanalysis results

#90

Post by simianwords »

jiggawatts wrote: post #54
simonw wrote: post #6

This is good: > If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that . The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...] > On the other hand : if y…

My mental model is this: There is a vast ocean of human knowledge, far beyond the capacity of any human brain, even within specialised fields. Books helped "plug the gaps" in our knowledge, increasing the scope that a single human mind can encompass. Web search engines did the same thing, but more and faster. LLMs are like search engines on steroids, essentially a research librarian that operates at 1,000x human spee…

Well put. I just think humans do this at a higher better level and not qualitatively different.
Post reply on HN