Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S1: A $6 R1 competitor?

timkellogg.me

Post reply on HN
Print view View original

161–170 of 430 posts

Re: S1: A $6 R1 competitor?

#161

Post by buyucu »

bloomingkales wrote: post #75

This thing that people are calling “reasoning” is more like rendering to me really, or multi pass rendering. We’re just refining the render, there’s no reasoning involved.

Yes.

Before LLMs we had N-Gram language models. Many tasks like speech recognition worked as beach search in the graph defined by the ngram language model. You could easily get huge accuracy gains simply by pruning your beam less.

s1 reminds of this. You can always trade off latency for accuracy. Given these LLMs are much more complex than good old N-Grams, we're just discovering how to do this trade.

Re: S1: A $6 R1 competitor?

#162

Post by UncleEntity »

khazhoux wrote: post #143

I have a bunch of questions, would love for anyone to explain these basics: * The $5M DeepSeek-R1 (and now this cheap $6 R1) are both based on very expensive oracles (if we believe DeepSeek-R1 queried OpenAI's model). If these are improvements on existing models, why is this being reported as decimating training costs? Isn't fine-tuning already a cheap way to optimize? (maybe not as effective, but still) * The R1 pap…

> If these are improvements on existing models, why is this being reported as decimating training costs?

Because that's what gets the clicks...

Saying they spent a boatload of money on the initial training + iteration + final fine-tuning isn't as headline grabbing as "$5 million trained AI beats the pants off the 'mericans".

Re: S1: A $6 R1 competitor?

#163

Post by robrenaud »

> "Note that this s1 dataset is distillation. Every example is a thought trace generated by another model, Qwen2.5"

The traces are generated by Gemini Flash Thinking.

8 hours of H100 is probably more like $24 if you want any kind of reliability, rather than $6.

Re: S1: A $6 R1 competitor?

#164

Post by LeifCarrotson »

tyfon wrote: post #139
gessha wrote: post #56

Earlier quoted context omitted.

This is pure speculation on my part but I think at some point a company's valuation became tied to how big their compute is so everybody jumped on the bandwagon.

I don't think you need to speculate too hard. On CNBC they are not tracking revenue, profits or technical breakthroughs, but how much the big companies are spending (on gpus). That's the metric!

I probably don't have to repeat it, but this is a perfect example of Goodhart's Law: when a metric is used as a target, it loses its effectiveness as a metric.

If you were a reporter who didn't necessarily understand how to value a particular algorithm or training operation, but you wanted a simple number to compare the amount of work OpenAI vs. Google vs Facebook are putting into their models, yeah, it makes sense. How many petaflops their datacenters are churning through in aggregate is probably correlated to the thing you're trying to understand. And it's probably easier to look at their financials and correlate how much they've spent on GPUs to how many petaflops of compute they need.

But when your investors are giving you more money based on how well they perceive you're doing, and their perception is not an oracle but is instead directly based on how much money you're spending... the GPUs don't actually need to do anything other than make number go up.

Re: S1: A $6 R1 competitor?

#165

Post by bloomingkales »

buyucu wrote: post #161
bloomingkales wrote: post #75

This thing that people are calling “reasoning” is more like rendering to me really, or multi pass rendering. We’re just refining the render, there’s no reasoning involved.

Yes. Before LLMs we had N-Gram language models. Many tasks like speech recognition worked as beach search in the graph defined by the ngram language model. You could easily get huge accuracy gains simply by pruning your beam less. s1 reminds of this. You can always trade off latency for accuracy. Given these LLMs are much more complex than good old N-Grams, we're just discovering how to do this trade.

Let me carry that concept, “learning to do this trade”, it’s a new trade.

I don’t believe computer science has the algorithms to handle this new paradigm. Everything was about sequential deterministic outputs, and clever ways to do it fast. This stuff is useless at the moment. We need new thinkers on how to not think sequentially or how not to think about the universe in such a small way.

Verifying input/output pairs is the old way. We need to understand differently going forward.

Re: S1: A $6 R1 competitor?

#166

Post by UncleEntity »

dotancohen wrote: post #48
yapyap wrote: post #8

> If you believe that AI development is a prime national security advantage, then you absolutely should want even more money poured into AI development, to make it go even faster. This, this is the problem for me with people deep in AI. They think it’s the end all be all for everything. They have the vision of the ‘AI’ they’ve seen in movies in mind, see the current ‘AI’ being used and to them it’s basically almost t…

> Real AI, artificial intelligence, is a fever dream. This is machine learning except the machines are bigger than ever before. There is no intellect. That sounds to me like dismissing the idea that a Russian SSBN might cross the Pacific and nuke Los Angeles because "submarines can't swim". Even if the machine learning isn't really intelligent, it is still capable of performing IF..THEN..ELSE operations, which could…

One could hope...

Even at the height of the Cold War there was always a human between and .

--edit--

...which has me wondering if a president even has the constitutional authority to destroy the entire planet and if one could interpret their command as a 'lawful order'. Makes one think.

Re: S1: A $6 R1 competitor?

#167

Post by sfink »

jebarker wrote: post #66

S1 (and R1 tbh) has a bad smell to me or at least points towards an inefficiency. It's incredible that a tiny number of samples and some inserted tokens can have such a huge effect on model behavior. I bet that we'll see a way to have the network learn and "emerge" these capabilities during pre-training. We probably just need to look beyond the GPT objective.

I agree, but LLMs in general have a horrendously bad smell in terms of efficiency. s1 and r1 are just proving it.

The models' latent spaces are insanely large. The vast, vast majority pretty much has to be irrelevant and useless, it's just that the training commandeers random fragments of that space to link up the logic they need and it's really hard to know which of the weights are useless, which are useful but interchangeable with other weights, and which are truly load-bearing. You could probably find out easily by testing the model against every possible thing you ever might want it to do, just as soon as someone gets around to enumerating that non-enumerable collection of tasks.

These bogus tokens kind of demonstrate that the models are sort of desperate to escape the limitations imposed by the limited processing they're allowed to do -- they'll take advantage of thinking time even when it's provided in the silliest manner possible. It's amazing what you can live if it's all you have!

(Apologies for the extended anthropomorphizing.)

Re: S1: A $6 R1 competitor?

#168

Post by nico »

> Why did it cost only $6? Because they used a small model and hardly any data.

> After sifting their dataset of 56K examples down to just the best 1K, they found that the core 1K is all that’s needed to achieve o1-preview performance on a 32B model. Adding data didn’t raise performance at all.

> 32B is a small model, I can run that on my laptop. They used 16 NVIDIA H100s for 26 minutes per training run, that equates to around $6.

Re: S1: A $6 R1 competitor?

#169

Post by deadbabe »

swiftcoder wrote: post #22

> having 10,000 H100s just means that you can do 625 times more experiments than s1 did I think the ball is very much in their court to demonstrate they actually are using their massive compute in such a productive fashion. My BigTech experience would tend to suggest that frugality went out the window the day the valuation took off, and they are in fact just burning compute for little gain, because why not...

For starters every employee has an H100 under their desk.

Re: S1: A $6 R1 competitor?

#170

Post by baxtr »

littlestymaar wrote: post #138
baxtr wrote: post #73

Earlier quoted context omitted.

If so wouldn’t it be the first time in history when more processing power is not used? In my experience CPU/GPU power is used up as much as possible. Increased efficiency just leads to more demand.

I think you're missing the point: H100 isn't going to remain useful for a long time, would you consider Tesla or Pascal graphic cards a collateral? That's what those H100 will look like in just a few years.

Not sure I do tbh.

Any asset depreciates over time. But they usually get replaced.

My 286 was replaced by a faster 386 and that by an even faster 468.

I’m sure you see a naming pattern there.

Post reply on HN