Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S1: A $6 R1 competitor?

timkellogg.me

Post reply on HN
Print view View original

141–150 of 430 posts

Re: S1: A $6 R1 competitor?

#141

Post by Mistletoe »

tyfon wrote: post #139
gessha wrote: post #56

Earlier quoted context omitted.

This is pure speculation on my part but I think at some point a company's valuation became tied to how big their compute is so everybody jumped on the bandwagon.

I don't think you need to speculate too hard. On CNBC they are not tracking revenue, profits or technical breakthroughs, but how much the big companies are spending (on gpus). That's the metric!

This feels like one of those stats they show from 1929 and everyone is like “and they didn’t know they were in a bubble?”

Re: S1: A $6 R1 competitor?

#143

Post by khazhoux »

I have a bunch of questions, would love for anyone to explain these basics:

* The $5M DeepSeek-R1 (and now this cheap $6 R1) are both based on very expensive oracles (if we believe DeepSeek-R1 queried OpenAI's model). If these are improvements on existing models, why is this being reported as decimating training costs? Isn't fine-tuning already a cheap way to optimize? (maybe not as effective, but still)

* The R1 paper talks about improving one simple game - Countdown. But the original models are "magic" because they can solve a nearly uncountable number of problems and scenarios. How does the DeepSeek / R1 approach scale to the same gigantic scale?

* Phrased another way, my understanding is that these techniques are using existing models as black-box oracles. If so, how many millions/billions/trillions of queries must be probed to replicate and improve the original dataset?

* Is anything known about the training datasets used by DeepSeek? OpenAI used presumably every scraped dataset they could get their hands on. Did DS do the same?

Re: S1: A $6 R1 competitor?

#144

Post by jjk7 »

pona-a wrote: post #53

If chain of thought acts as a scratch buffer by providing the model more temporary "layers" to process the text, I wonder if making this buffer a separate context with its own separate FNN and attention would make sense; in essence, there's a macroprocess of "reasoning" that takes unbounded time to complete, and then there's a microprocess of describing this incomprehensible stream of embedding vectors in natural lan…

Comments on a google doc? Nesting in social media comments?

Seems like similar concepts. I think there is some potential to improving how LLMs improve and further their own reasoning lines, but I'm no AI mage.

Re: S1: A $6 R1 competitor?

#145

Post by jebarker »

whimsicalism wrote: post #101
jebarker wrote: post #71

Earlier quoted context omitted.

The same idea is in both the R1 and S1 papers ( tokens are used similarly). Basically they're using special tokens to mark in the prompt where the LLM should think more/revise the previous response. This can be repeated many times until some stop criteria occurs. S1 manually inserts these with heuristics, R1 learns the placement through RL I think.

? theyre not special tokens really

i'm not actually sure whether they're special tokens in the sense of being in the vocabulary

Re: S1: A $6 R1 competitor?

#146

Post by whimsicalism »

jebarker wrote: post #145
whimsicalism wrote: post #101

Earlier quoted context omitted.

? theyre not special tokens really

i'm not actually sure whether they're special tokens in the sense of being in the vocabulary

might be i think "wait" is tokenized like any other in the pretraining

Re: S1: A $6 R1 competitor?

#147

Post by sdenton4 »

throwup238 wrote: post #115
adrianN wrote: post #87

Earlier quoted context omitted.

They have a fairly limited lifetime even if progress stands still.

Last I checked AWS 1-year reserve pricing for an 8x H100 box more than pays for the capital cost of the whole box, power, and NVIDIA enterprise license, with thousands left over for profit. On demand pricing is even worse. For cloud providers these things pay for themselves quickly and print cash afterwards. Even the bargain basement $2/GPU/hour pays it off in under two years.

Labor! You need it to turn the bill of sale into a data center and keep it running. The bargain basement would be even cheaper otherwise...

Re: S1: A $6 R1 competitor?

#148

Post by gom_jabbar »

cubefox wrote: post #137
mtrovo wrote: post #133

I found the discussion around inference scaling with the 'Wait' hack so surreal. The fact such an ingeniously simple method can impact performance makes me wonder how many low-hanging fruit we're still missing. So weird to think that improvements on a branch of computer science is boiling down to conjuring the right incantation words, how you even change your mindset to start thinking this way?

Now imagine where we are in 12 months from now. This article from February 5 2025 will feel quaint by then. The acceleration keeps increasing. It seems likely we will soon have recursive self-improving AI -- reasoning models which do AI research. This will accelerate the rate of acceleration itself. It sounds stupid to say it, but yes, the singularity is near. Vastly superhuman AI now seems to arrive within the next…

Yes, and Accelerationism predicted this development back in the 1990s, perhaps most prominently in the opening lines of Nick Land's Meltdown (1994) text:

  [[ ]] The story goes like this: Earth is captured by a technocapital singularity as renaissance rationalization and oceanic navigation lock into commoditization take-off. Logistically accelerating techno-economic interactivity crumbles social order in auto-sophisticating machine runaway. As markets learn to manufacture intelligence, politics modernizes, upgrades paranoia, and tries to get a grip.
> reasoning models which do AI research

In the introduction to my research project on Accelerationism [0], I write:

  Faced with the acceleration of progress in Artificial Intelligence (AI) — with AI agents now automating AI research and development —, Accelerationism no longer seems like an abstract philosophy producing empty hyperstitional hype, but like a sober description of reality. The failed 2023 memorandum to stop AI development on systems more powerful than OpenAI's ChatGPT-4 perfectly illustrates the phenomenological aspects of Accelerationism: "To be rushed by the phenomenon, to the point of terminal institutional paralysis, is the phenomenon." [1]
At the current rate of acceleration, if you don't write hyperstitionally, your texts are dead on arrival.

[0] https://retrochronic.com/

[1] Nick Land (2017). A Quick-and-Dirty Introduction to Accelerationism in Jacobite Magazine.

Re: S1: A $6 R1 competitor?

#149

Post by spamizbad »

sixothree wrote: post #82
jxdxbx wrote: post #69

Earlier quoted context omitted.

I hear this a lot, but what the hell. It's still computer chips. They depreciate. Short supply won't last forever. Hell, GPUs burn out. It seems like using ice sculptures as collateral, and then spring comes.

Year over year gains in computing continue to slow. I think we keep forgetting that when talking about these things as assets. The thing controlling their value is the supply which is tightly controlled like diamonds.

> Year over year gains in computing continue to slow.

This isn't true in the AI chip space (yet). And so much of this isn't just about compute but about the memory.

Re: S1: A $6 R1 competitor?

#150

Post by ijidak »

littlestymaar wrote: post #138
baxtr wrote: post #73

Earlier quoted context omitted.

If so wouldn’t it be the first time in history when more processing power is not used? In my experience CPU/GPU power is used up as much as possible. Increased efficiency just leads to more demand.

I think you're missing the point: H100 isn't going to remain useful for a long time, would you consider Tesla or Pascal graphic cards a collateral? That's what those H100 will look like in just a few years.

Yeah, exactly! I've got some 286, 386, and 486 CPUs that I want to claim as collateral!
Post reply on HN