S1: A $6 R1 competitor? • Hacker News Board

Re: S1: A $6 R1 competitor?

#121

Post by Aperocky » Wed, Feb 05, 2025, 4:11 PM UTC

For all the hype about thinking models, this feels much like compression in terms of information theory instead of a "takeoff" scenario.

There are a finite amount of information stored in any large model, the models are really good at presenting the correct information back, and adding thinking blocks made the models even better at doing that. But there is a cap to that.

Just like how you can compress a file by a lot, there is a theoretical maximum to the amount of compression before it starts becoming lossy. There is also a theoretical maximum of relevant information from a model regardless of how long it is forced to think.

Re: S1: A $6 R1 competitor?

#122

Post by [deleted] » Wed, Feb 05, 2025, 4:12 PM UTC

whimsicalism wrote: post #108

menaerus wrote: post #47

Earlier quoted context omitted.

You can choose to be somewhat ignorant of the current state in AI, about which I could also agree that at certain moments it appears totally overhyped, but the reality is that there hasn't been a bigger technology breakthrough probably in the last ~30 years. This is not "just" machine learning because we have never been able to do things which we are today and this is not only the result of better hardware. Better ha…

> the first one being from DeepMind in 2017 ? what paper are you talking about

[deleted]

Re: S1: A $6 R1 competitor?

#123

Post by amarcheschi » Wed, Feb 05, 2025, 4:15 PM UTC

fkyoureadthedoc wrote: post #107

amarcheschi wrote: post #16

Earlier quoted context omitted.

I couldn't agree more. If we're not talking about cyber war exclusively, such as finding and exploiting vulnerabilities, for the time being national security will still be based on traditional army. Just a few weeks ago, italy announced a 16bln€ plan to buy >1000 rheinmetall ifv vehicles. That alone would make italy's army one of the most equipped in Europe. I can't imagine what would happen with a 500$bln investment…

> I can't imagine what would happen with a 500$bln investment in defense,lol. The $90,000 bag of bushings becomes a $300,000 bag?

No, at least in italy we would create some gimmick consultant position for some ex politician and repay him/her for past favours /s

Re: S1: A $6 R1 competitor?

#124

Post by ecocentrik » Wed, Feb 05, 2025, 4:16 PM UTC

jxdxbx wrote: post #69

syntaxing wrote: post #61

Earlier quoted context omitted.

Matt Levine tangentially talked about this during his podcast this past Friday (or was it the one before?). It was a good way to value these companies according to their compute size since those chips are very valuable. At a minimum, the chips are an asset that acts as a collateral.

I hear this a lot, but what the hell. It's still computer chips. They depreciate. Short supply won't last forever. Hell, GPUs burn out. It seems like using ice sculptures as collateral, and then spring comes.

That is the wrong take. Depreciated and burned out chips are replaced and a total compute value is typically increased over time. Efficiency gains are also calculated and projected over time. Seasons are inevitable and cyclical. Spring might be here but winter is coming.

Re: S1: A $6 R1 competitor?

#125

Post by qup » Wed, Feb 05, 2025, 4:27 PM UTC

ALittleLight wrote: post #119

At 6 dollars per run, I'm tempted to try to figure out how to replicate this. I'd like to try some alternatives to "wait" - e.g. "double checking..." Or write my own chains of thought.

Like the ones they tested?

Re: S1: A $6 R1 competitor?

#126

Post by withinboredom » Wed, Feb 05, 2025, 4:28 PM UTC

mettamage wrote: post #93

davely wrote: post #67

Earlier quoted context omitted.

I think we need to wait for someone to convert it into a GGUF file format. However, once that happens, you can run it (and any GGUF model) from Hugging Face![0] [0] https://huggingface.co/docs/hub/en/ollama

So this? https://huggingface.co/brittlewis12/s1-32B-GGUF

oh god, this is terrible!

I just said "Hello!" and it went off the rails.

Re: S1: A $6 R1 competitor?

#127

Post by psadri » Wed, Feb 05, 2025, 4:36 PM UTC

bloomingkales wrote: post #113

pona-a wrote: post #53

If chain of thought acts as a scratch buffer by providing the model more temporary "layers" to process the text, I wonder if making this buffer a separate context with its own separate FNN and attention would make sense; in essence, there's a macroprocess of "reasoning" that takes unbounded time to complete, and then there's a microprocess of describing this incomprehensible stream of embedding vectors in natural lan…

Once we train models on the chain of thought outputs, next token prediction can solve the halting problem for us (eg, this chain of thinking matches this other chain of thinking).

I think that is how human brains work. When we practice, at first we have to be deliberate (thinking slow). Then we “learn” from our own experience and it becomes muscle memory (thinking fast). Of course, it increases the odds we are wrong.

Re: S1: A $6 R1 competitor?

#128

Post by psadri » Wed, Feb 05, 2025, 4:38 PM UTC

Aperocky wrote: post #121

For all the hype about thinking models, this feels much like compression in terms of information theory instead of a "takeoff" scenario. There are a finite amount of information stored in any large model, the models are really good at presenting the correct information back, and adding thinking blocks made the models even better at doing that. But there is a cap to that. Just like how you can compress a file by a lot…

I think an interesting avenue to explore is creating abstractions and analogies. If a model can take a novel situation and create an analogy to one that it is familiar with, it would expand its “reasoning” capabilities beyond its training data.

Re: S1: A $6 R1 competitor?

#129

Post by hidelooktropic » Wed, Feb 05, 2025, 4:40 PM UTC

> I doubt that OpenAI has a realistic path to preventing or even detecting distealing outside of simply not releasing models.

Couldn't they just start hiding the thinking portion?

It would be easy for them to do this. Currently, they already provide one sentence summaries for each step of the thinking I think users would be fine or at least stay if it were changed to provide only that.

Re: S1: A $6 R1 competitor?

#130

Post by bloomingkales » Wed, Feb 05, 2025, 4:45 PM UTC

psadri wrote: post #127

bloomingkales wrote: post #113

Earlier quoted context omitted.

Once we train models on the chain of thought outputs, next token prediction can solve the halting problem for us (eg, this chain of thinking matches this other chain of thinking).

I think that is how human brains work. When we practice, at first we have to be deliberate (thinking slow). Then we “learn” from our own experience and it becomes muscle memory (thinking fast). Of course, it increases the odds we are wrong.

Or worse, we incorrectly overweight the wrong chain of thinking to an irrelevant output (but pragmatically useful output), at scale.

For example, xenophobia as a response to economic hardship is the wrong chain of thinking embedded in the larger zeitgeist.