Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S1: A $6 R1 competitor?

timkellogg.me

Post reply on HN
Print view View original

271–280 of 430 posts

Re: S1: A $6 R1 competitor?

#271

Post by palmotea »

lodovic wrote: post #268
immibis wrote: post #229

Earlier quoted context omitted.

Those people will simply be surplus to requirements. They'll be left alone as long as they don't get in the way of the ruling class, and disposed of if they do. As usual in history.

That's a fallacy. You can't have an advanced economy with most people sitting on the side. Money needs to keep flowing. If all that remains of the economy consists of a few datacenters talking to each other, how can the ruling class profit off that?

> You can't have an advanced economy with most people sitting on the side.

If AI lives up to the hype, that will become possible.

> If all that remains of the economy consists of a few datacenters talking to each other, how can the ruling class profit off that?

I don't think it would be that. There'd also be power generation, manufacturing, mining, and construction, etc.; but all extremely automated. If you get to truly extreme levels of wealth concentration, things would shift out of our capitalist market system model, and concepts like "profit" would become anachronisms.

It actually might kinda look like the "economy" of Starcraft: you gather resources, decide what to build with them, and order it all around according to your whim. There will be a handful of guys playing, and everyone else will be a NPC.

Re: S1: A $6 R1 competitor?

#272

Post by cakealert »

pizza wrote: post #260
cakealert wrote: post #251

Earlier quoted context omitted.

RL provides very poor training signal for deep learning, an order of magnitude or more worse than supervised learning. Better than nothing of course. What the OP suggested is similar to training a transformer from scratch using RL (ie. no training tokens) towards an objective of steering a pretrained LLM to produce human readable output. It will probably not even converge, and if it does it would take immense compute…

In the case of supervised problem domains, you implicitly make a decision about what is signal, and what is noise, and sure, in that closed setting, supervised learning is much more sample efficient. But I think what we're learning now is that with strong enough base models, 'aha' moments in RL training show that it might be possible to essentially 'squeeze out signal from language itself', giving you far greater bre…

With a pretrained LLM most of the work is done. RL just steers the model into a 'thinking' mode. There is enough signal for that to work and for the inefficiency to not matter.

The downside is that you are limiting the model to think in the same language it outputs. An argument could be made that this is not how all humans think. I know that I rarely think in language or even images, just concepts (probably isn't even the right word) mix and transform and often I don't even bother to make the transformation to language at the end, just action.

Re: S1: A $6 R1 competitor?

#274

Post by menaerus »

mitchelld wrote: post #219
menaerus wrote: post #172

Earlier quoted context omitted.

What line of thinking you're referring to? Transformers were aimed to solve the "context" problem and authors, being aware that RNNs don't scale at all neither do they solve that particular problem, had to come up with the algorithm that overcomes both of those issues. It turned out that the self-attention compute-scale was the crucial ingredient to solve the problem, something that RNNs were totally incapable of. Th…

> This is not "just" machine learning because we have never been able to do things which we are today and this is not only the result of better hardware. Better hardware is actually a byproduct. Why build a PFLOPS GPU when there is nothing that can utilize it? This is the line of thinking I'm referring to. The "context" problem had already been somewhat solved. The attention mechanism existed prior to Transformers an…

> innovation of the architecture was making it computation efficient to train.

and

> researchers demonstrated that simply scaling up training with more data yielded better models

and

> The fact that hardware was then optimized for these for these architectures only reinforces this point.

and

> All the papers discussing scaling laws point to the same thing, simply using more compute and data yields better results.

is what I am saying as well. I read the majority of those papers so this is all very known to me but I am perhaps writing it down in a more condensed format so that other readers that are light on the topic can pick the idea easier.

> A majority of the improvement from GPT-2 and GPT-4 was simply training on a much larger scale. That was enabled by better hardware and lots of it.

Ok, I see your point and the conclusion here is what we disagree with. You say that the innovation was simply enabled by the better hardware whereas I say that that better hardware wouldn't have its place if there hadn't been a great innovation in the algorithm itself. I don't think it's fair to say that the innovation is driven by the NVidia chips.

I guess my point, simplistically saying, is if we had a lousy algorithm, new hardware wouldn't mean anything without rethinking or rewriting the algorithm. And with the transformers, this definitely hadn't been the case. There had been plenty of optimizations throughout the years in order to better utilize the HW (e.g. flash-attention) but the architecture of transformers remained more or less the same.

Re: S1: A $6 R1 competitor?

#275

Post by ChymeraXYZ »

brookst wrote: post #262
gessha wrote: post #204

Earlier quoted context omitted.

Your example is somewhat inadequate. We _fundamentally_ don’t understand how deep learning systems works in the sense that they are more or less black boxes that we train and evaluate. Innovations in ML are a whole bunch of wizards with big stacks of money changing “Hmm” to “Wait” and seeing what happens. Would a different sampler help you? I dunno, try it. Would a smaller dataset help? I dunno, try it. Would trainin…

Isn't that just scale? Even small LLMs have more parts than any car. LLMs are more analogous to economics, psychology, politics -- it is possible there's a core science with explicability, but the systems are so complex that even defining the question is hard.

Could be, but it does not change the fact that we do not understand them as of now.

Re: S1: A $6 R1 competitor?

#276

Post by ascorbic »

lionkor wrote: post #187
ascorbic wrote: post #185

Earlier quoted context omitted.

I've noticed that R1 says "Wait," a lot in its reasoning. I wonder if there's something inherently special in that token.

Semantically, wait is a bit of a stop-and-breathe point. Consider the text: I think I'll go swimming today. Wait, ___ what comes next? Well, not something that would usually follow without the word "wait", probably something entirely orthogonal that impacts the earlier sentence in some fundamental way, like: Wait, I need to help my dad.

Yes, R1 seems to mostly use it like that. It's either to signal a problem with its previous reasoning, or if it's thought of a better approach. In coding it's often something like "this API won't work here" or "there's a simpler way to do this".

Re: S1: A $6 R1 competitor?

#277

Post by advael »

I'm strictly speaking never going to think of model distillation as "stealing." It goes against the spirit of scientific research, and besides every tech company has lost my permission to define what I think of as theft forever

Re: S1: A $6 R1 competitor?

#278

Post by ascorbic »

katzenversteher wrote: post #265
ascorbic wrote: post #185

Earlier quoted context omitted.

I've noticed that R1 says "Wait," a lot in its reasoning. I wonder if there's something inherently special in that token.

I bet a token like "sh t!", "f* " or "damn!" would have the same or even stronger effect but the LLM creators would not like to have the users read them

Maybe, but it doesn't just use it to signify that it's made a mistake. It also uses it in a positive way, such as it's had a lightbulb moment. Of course some people use expletives in the same way, but that would be less common than for mistakes.

Re: S1: A $6 R1 competitor?

#279

Post by pizza »

cakealert wrote: post #272
pizza wrote: post #260

Earlier quoted context omitted.

In the case of supervised problem domains, you implicitly make a decision about what is signal, and what is noise, and sure, in that closed setting, supervised learning is much more sample efficient. But I think what we're learning now is that with strong enough base models, 'aha' moments in RL training show that it might be possible to essentially 'squeeze out signal from language itself', giving you far greater bre…

With a pretrained LLM most of the work is done. RL just steers the model into a 'thinking' mode. There is enough signal for that to work and for the inefficiency to not matter. The downside is that you are limiting the model to think in the same language it outputs. An argument could be made that this is not how all humans think. I know that I rarely think in language or even images, just concepts (probably isn't eve…

I strongly agree; in fact I think what best matches the thought process is something like the multiset tree/forest workspace approach as suggested by Marcolli, Chomsky, and Berwick - a Hopf algebra that can be externalized into (non-planar) embeddings of linearized strings, or alternately into semantic manifolds.

Re: S1: A $6 R1 competitor?

#280

Post by Nurbek-F »

gorgoiler wrote: post #270

This feels just like telling a constraint satisfaction engine to backtrack and find a more optimal route through the graph. We saw this 25 years ago with engines like PROVERB doing directed backtracking, and with adversarial planning when automating competitive games. Why would you control the inference at the token level? Wouldn’t the more obvious (and technically superior) place to control repeat analysis of the op…

Totally agreed this is not a solution we are looking for, in fact this is the only solution we have in our hands right now. It's a good step forward.
Post reply on HN