Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S1: A $6 R1 competitor?

timkellogg.me

Post reply on HN
Print view View original

301–310 of 430 posts

Re: S1: A $6 R1 competitor?

#301

Post by coder543 »

3abiton wrote: post #263
ZeljkoS wrote: post #228

Earlier quoted context omitted.

We have a partial understanding of why distillation works—it is explained by The Lottery Ticket Hypothesis ( https://arxiv.org/abs/1803.03635 ). But if I am understanding correctly, that doesn't mean you can train a smaller network from scratch. You need a lot of randomness in the initial large network, for some neurons to have "winning" states. Then you can distill those winning subsystems to a smaller network. Note…

So more 'mature' models might arise in the near future with less params and better benchmarks?

That's been happening consistently for over a year now. Small models today are better than big models from a year or two ago.

Re: S1: A $6 R1 competitor?

#302

Post by homarp »

umeshunni wrote: post #269
MR4D wrote: post #259

Earlier quoted context omitted.

I like the analogy of compression, in that a distilled model of an LLM is like a JPEG of a photo. Pretty good, maybe very good, but still lossy. The question I hear you raising seems to be along the lines of, can we use a new compression method to get better resolution (reproducibility of the original) in a much smaller size.

> in that a distilled model of an LLM is like a JPEG of a photo That's an interesting analogy, because I've always thought of the hidden states (and weights and biases) of an LLMs as a compressed version of the training data.

hence https://www.newyorker.com/tech/annals-of-technology/chatgpt-... (by Ted Chiang)

(discussed here: https://news.ycombinator.com/item?id=34724477 )

Re: S1: A $6 R1 competitor?

#303

Post by ozgune »

mtrovo wrote: post #133

I found the discussion around inference scaling with the 'Wait' hack so surreal. The fact such an ingeniously simple method can impact performance makes me wonder how many low-hanging fruit we're still missing. So weird to think that improvements on a branch of computer science is boiling down to conjuring the right incantation words, how you even change your mindset to start thinking this way?

Agreed. Here are three things that I find surreal about the s1 paper.

(1) The abstract changed how I thought about this domain (advanced reasoning models). The only other paper that did that for me was the "Memory Resource Management in VMware ESX Server". And that paper got published 23 years ago.

(2) The model, data, and code are open source at https://github.com/simplescaling/s1. With this, you can start training your own advanced reasoning models. All you need is a thousand well-curated questions with reasoning steps.

(3) More than half the references in the paper are from 2024 and Jan 2025. Just look at the paper's first page. https://arxiv.org/pdf/2501.19393 In which other field do you see this?

Re: S1: A $6 R1 competitor?

#305

Post by latexr »

mangoman wrote: post #234

From the S1 paper: > Second, we develop budget forcing to control test-time compute by forcefully terminating the model's thinking process or lengthening it by appending "Wait" multiple times to the model's generation when it tries to end I'm feeling proud of myself that I had the crux of the same idea almost 6 months ago before reasoning models came out (and a bit disappointed that I didn't take this idea further!).…

> and a bit disappointed that I didn't take this idea further!

Don’t be, that’s pretty common.

https://en.wikipedia.org/wiki/Multiple_discovery

Re: S1: A $6 R1 competitor?

#307

Post by latexr »

falcor84 wrote: post #103
incrudible wrote: post #94

Hmmm, 1 + 1 equals 3. Alternatively, 1 + 1 equals -3. Wait, actually 1 + 1 equals 1.

As one with teaching experience, the idea of asking a student "are you sure about that?" is to get them to think more deeply rather than just blurting a response. It doesn't always work, but it generally does.

It works because the question itself is a hint born of knowledge. “Are you sure about that” is a polite way to say “that answer is wrong, try again”. Students know that, so instead of doubling down will redo their work with the assumption they made a mistake. It is much rarer to ask the question when the answer is correct, and in fact doing so is likely to upset the learner because they had to redo the work for no reason.

If you want a true comparison, start asking that question every time and then compare. My hypothesis is students would start ignoring the prompt and answering “yes” every time to get on with it.

Re: S1: A $6 R1 competitor?

#308

Post by gessha »

raducu wrote: post #287
gessha wrote: post #204

Earlier quoted context omitted.

Your example is somewhat inadequate. We _fundamentally_ don’t understand how deep learning systems works in the sense that they are more or less black boxes that we train and evaluate. Innovations in ML are a whole bunch of wizards with big stacks of money changing “Hmm” to “Wait” and seeing what happens. Would a different sampler help you? I dunno, try it. Would a smaller dataset help? I dunno, try it. Would trainin…

> _fundamentally_ don’t understand how deep learning systems works. It's like saying we don't understand how quantum chromodynamics works. Very few people do, and it's the kind of knowledge not easily distilled for the masses in an easily digestible in a popsci way. Look into how older CNNs work -- we have very good visual/accesible/popsci materials on how they work. I'm sure we'll have that for LLM but it's not wort…

As a person who has trained a number of computer vision deep networks, I can tell you that we have some cool-looking visualizations on how lower layers work but no idea how later layers work. The intuition is built over training numerous networks and trying different hyperparameters, data shuffling, activations, etc. it’s absolutely brutal over here. If the theory was there, people like Karpathy who have great teacher vibes would’ve explained it for the mortal grad students or enthusiast tinkerers.

> The kind of progress being made leads me to believe there absolutely ARE people who absolutely know how the LLMs work and they're not just a bunch of monkeys randomly throwing things at GPUs and seeing what sticks

I say this less as an authoritative voice but more as an amused insider: Spend a week with some ML grad students and you will get a chuckle whenever somebody says we’re not some monkeys throwing things at GPUs.

Re: S1: A $6 R1 competitor?

#309

Post by gessha »

brookst wrote: post #262
gessha wrote: post #204

Earlier quoted context omitted.

Your example is somewhat inadequate. We _fundamentally_ don’t understand how deep learning systems works in the sense that they are more or less black boxes that we train and evaluate. Innovations in ML are a whole bunch of wizards with big stacks of money changing “Hmm” to “Wait” and seeing what happens. Would a different sampler help you? I dunno, try it. Would a smaller dataset help? I dunno, try it. Would trainin…

Isn't that just scale? Even small LLMs have more parts than any car. LLMs are more analogous to economics, psychology, politics -- it is possible there's a core science with explicability, but the systems are so complex that even defining the question is hard.

You can make a bigger ICE engine (like a container ship engine) and still understand how the whole thing works. Maybe there’s more parts moving but it still has the structure of an ICE engine.

With neural networks big or small, we got no clue what’s going on. You can observe the whole system, from the weights and biases, to the activations, gradients, etc and still get nothing.

On the other hand, one of the reasons why economics, psychology and politics are hard is because we can’t open up people’s heads and define and measure what they’re thinking.

Re: S1: A $6 R1 competitor?

#310

Post by salemba »

raducu wrote: post #281
3abiton wrote: post #263

Earlier quoted context omitted.

So more 'mature' models might arise in the near future with less params and better benchmarks?

"Better", but not better than the model they were distilled from, at least that's how I understand it.

I think this is how the "child brain" works too. The better the parents and the environement are, the better the child evolution is :)
Post reply on HN