Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S1: A $6 R1 competitor?

timkellogg.me

Post reply on HN
Print view View original

171–180 of 430 posts

Re: S1: A $6 R1 competitor?

#171

Post by nico »

> In s1, when the LLM tries to stop thinking with "", they force it to keep going by replacing it with "Wait". It’ll then begin to second guess and double check its answer. They do this to trim or extend thinking time (trimming is just abruptly inserting "")

I know some are really opposed to anthropomorphizing here, but this feels eerily similar to the way humans work, ie. if you just dedicate more time to analyzing and thinking about the task, you are more likely to find a better solution

It also feels analogous to navigating a tree, the more time you have to explore the nodes, the bigger the space you'll have covered, hence higher chance of getting a more optimal solution

At the same time, if you have "better intuition" (better training?), you might be able to find a good solution faster, without needing to think too much about it

Re: S1: A $6 R1 competitor?

#172

Post by menaerus »

mitchelld wrote: post #134
menaerus wrote: post #47

Earlier quoted context omitted.

You can choose to be somewhat ignorant of the current state in AI, about which I could also agree that at certain moments it appears totally overhyped, but the reality is that there hasn't been a bigger technology breakthrough probably in the last ~30 years. This is not "just" machine learning because we have never been able to do things which we are today and this is not only the result of better hardware. Better ha…

This line of thinking doesn't really correspond to the reason Transformers were developed in the first place, which was to better utilize how GPUs do computation. RNNs were too slow to train at scale because you had to sequentially compute the time steps, Transformers (with masking) can run the input through in a single pass. It is worth noting that the first "LLM" you referring to was only 300M parameters, but even…

What line of thinking you're referring to?

Transformers were aimed to solve the "context" problem and authors, being aware that RNNs don't scale at all neither do they solve that particular problem, had to come up with the algorithm that overcomes both of those issues. It turned out that the self-attention compute-scale was the crucial ingredient to solve the problem, something that RNNs were totally incapable of.

They modeled the algorithm to run on the hardware they had at that time available but hardware developed afterwards was a direct consequence, or how I called it a byproduct, of transformers proving themselves to be able to continuously scale. Had that not be true, we wouldn't have all those iterations of NVidia chips.

So, although one could say that the NVidia chip design is what enabled the transformers success, one could also say that we wouldn't have those chips if transformers didn't prove themselves to be so damn efficient. And I'm inclined to think the latter.

Re: S1: A $6 R1 competitor?

#173

Post by janalsncm »

I think a lot of people in the ML community were excited for Noam Brown to lead the O series at OpenAI because intuitively, a lot of reasoning problems are highly nonlinear i.e. they have a tree-like structure. So some kind of MCTS would work well. O1/O3 don’t seem to use this, and DeepSeek explicitly mentioned difficulties training such a model.

However, I think this is coming. DeepSeek mentioned it was hard to learn a value model for MCTS from scratch, but this doesn’t mean we couldn’t seed it with some annotated data.

Re: S1: A $6 R1 competitor?

#174

Post by willglynn »

UncleEntity wrote: post #166
dotancohen wrote: post #48

Earlier quoted context omitted.

> Real AI, artificial intelligence, is a fever dream. This is machine learning except the machines are bigger than ever before. There is no intellect. That sounds to me like dismissing the idea that a Russian SSBN might cross the Pacific and nuke Los Angeles because "submarines can't swim". Even if the machine learning isn't really intelligent, it is still capable of performing IF..THEN..ELSE operations, which could…

One could hope... Even at the height of the Cold War there was always a human between and . --edit-- ...which has me wondering if a president even has the constitutional authority to destroy the entire planet and if one could interpret their command as a 'lawful order'. Makes one think.

On the topic of fail-deadly nukes:

https://en.wikipedia.org/wiki/Dead_Hand

Re: S1: A $6 R1 competitor?

#175

Post by xg15 »

mtrovo wrote: post #133

I found the discussion around inference scaling with the 'Wait' hack so surreal. The fact such an ingeniously simple method can impact performance makes me wonder how many low-hanging fruit we're still missing. So weird to think that improvements on a branch of computer science is boiling down to conjuring the right incantation words, how you even change your mindset to start thinking this way?

I think the fact alone that distillation and quantization are techniques that can produce substantial improvements is a strong sign that we still have no real comprehensive understanding how the models work.

If we had, there would be no reason to train a model with more parameters than are strictly necessary to represent the space's semantic structure. But then it should be impossible for distilled models with less parameters to come close to the performance of the original model.

Yet this is what happens - the distilled or quantized models often come very close to the original model.

So I think there are still many low-hanging fruits to pick.

Re: S1: A $6 R1 competitor?

#176

Post by dghlsakjg »

syntaxing wrote: post #61
gessha wrote: post #56

Earlier quoted context omitted.

This is pure speculation on my part but I think at some point a company's valuation became tied to how big their compute is so everybody jumped on the bandwagon.

Matt Levine tangentially talked about this during his podcast this past Friday (or was it the one before?). It was a good way to value these companies according to their compute size since those chips are very valuable. At a minimum, the chips are an asset that acts as a collateral.

That's a great way to value a company that is going bankrupt.

But, I'm not going to value an operating construction company based on how many shovels or excavators they own. I'm going to want to see them putting those assets to productive use.

Re: S1: A $6 R1 competitor?

#177

Post by zaptrem »

robrenaud wrote: post #163

> "Note that this s1 dataset is distillation. Every example is a thought trace generated by another model, Qwen2.5" The traces are generated by Gemini Flash Thinking. 8 hours of H100 is probably more like $24 if you want any kind of reliability, rather than $6.

"You can train a SOTA LLM for $0.50" (as long as you're distilling a model that cost $500m into another pretrained model that cost $5m)

Re: S1: A $6 R1 competitor?

#178

Post by kgwgk »

baxtr wrote: post #170
littlestymaar wrote: post #138

Earlier quoted context omitted.

I think you're missing the point: H100 isn't going to remain useful for a long time, would you consider Tesla or Pascal graphic cards a collateral? That's what those H100 will look like in just a few years.

Not sure I do tbh. Any asset depreciates over time. But they usually get replaced. My 286 was replaced by a faster 386 and that by an even faster 468. I’m sure you see a naming pattern there.

> Any asset depreciates over time.

That's why "those chips are very valuable" is not necessarily a good way to value companies - and it isn't if they can extract the value from the chips before they become worthless.

> But they usually get replaced.

They usually produce enough income to cover depreciation so you actually have the cash to replace them.

Re: S1: A $6 R1 competitor?

#180

Post by insane-c0der »

janalsncm wrote: post #173

I think a lot of people in the ML community were excited for Noam Brown to lead the O series at OpenAI because intuitively, a lot of reasoning problems are highly nonlinear i.e. they have a tree-like structure. So some kind of MCTS would work well. O1/O3 don’t seem to use this, and DeepSeek explicitly mentioned difficulties training such a model. However, I think this is coming. DeepSeek mentioned it was hard to lear…

Do you have a reference for us to check? - "DeepSeek explicitly mentioned difficulties training such a model."
Post reply on HN