Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S1: A $6 R1 competitor?

timkellogg.me

Post reply on HN
Print view View original

281–290 of 430 posts

Re: S1: A $6 R1 competitor?

#281

Post by raducu »

3abiton wrote: post #263
ZeljkoS wrote: post #228

Earlier quoted context omitted.

We have a partial understanding of why distillation works—it is explained by The Lottery Ticket Hypothesis ( https://arxiv.org/abs/1803.03635 ). But if I am understanding correctly, that doesn't mean you can train a smaller network from scratch. You need a lot of randomness in the initial large network, for some neurons to have "winning" states. Then you can distill those winning subsystems to a smaller network. Note…

So more 'mature' models might arise in the near future with less params and better benchmarks?

"Better", but not better than the model they were distilled from, at least that's how I understand it.

Re: S1: A $6 R1 competitor?

#282

Post by _w1tm »

3abiton wrote: post #263
ZeljkoS wrote: post #228

Earlier quoted context omitted.

We have a partial understanding of why distillation works—it is explained by The Lottery Ticket Hypothesis ( https://arxiv.org/abs/1803.03635 ). But if I am understanding correctly, that doesn't mean you can train a smaller network from scratch. You need a lot of randomness in the initial large network, for some neurons to have "winning" states. Then you can distill those winning subsystems to a smaller network. Note…

So more 'mature' models might arise in the near future with less params and better benchmarks?

They might also be more biased and less able to adapt to new technology. Interesting times.

Re: S1: A $6 R1 competitor?

#283

Post by barrenko »

zoogeny wrote: post #196
pona-a wrote: post #53

If chain of thought acts as a scratch buffer by providing the model more temporary "layers" to process the text, I wonder if making this buffer a separate context with its own separate FNN and attention would make sense; in essence, there's a macroprocess of "reasoning" that takes unbounded time to complete, and then there's a microprocess of describing this incomprehensible stream of embedding vectors in natural lan…

I've had an idea since I was a kid which I can share. I was contemplating AI and consciousness generally, probably around the time I read "The Minds I". I reflected on the pop-psychology idea of consciousness and subconsciousness. I thought of each as an independent stream of tokens, like stream of consciousness poetry. But along the stream there were joining points between these two streams, points where the conscio…

Conscious as subconscious pretending not to be sunconscious, something like that, a thin wrapper. Crud makes sense.

Gels closely to buddhism, hell, all religions.

Re: S1: A $6 R1 competitor?

#284

Post by code_biologist »

gorgoiler wrote: post #270

This feels just like telling a constraint satisfaction engine to backtrack and find a more optimal route through the graph. We saw this 25 years ago with engines like PROVERB doing directed backtracking, and with adversarial planning when automating competitive games. Why would you control the inference at the token level? Wouldn’t the more obvious (and technically superior) place to control repeat analysis of the op…

What would a superior control approach be? It's not clear to me how to get an LLM to be an LLM if you're not doing stochastic next token prediction. Given that, the model itself is going to know best how to traverse its own concept space. The R1 chain of thought training encourages and develops exactly that capability. Still, you want that chain of thought to terminate and not navel gaze endlessly.

So how to externally prod it to think more when it does terminate? Replacing thought termination with a linguistic signifier of continued reasoning plus novel realization seems like a charmingly simple, principled, and general approach to continue to traverse concept space.

Re: S1: A $6 R1 competitor?

#285

Post by _w1tm »

palmotea wrote: post #212
zoogeny wrote: post #192

Earlier quoted context omitted.

This is something I have been suppressing since I don't want to become chicken little. Anyone who isn't terrified by the last 3 months probably doesn't really understand what is happening. I went from accepting I wouldn't see a true AI in my lifetime, to thinking it is possible before I die, to thinking it is possible in in the next decade, to thinking it is probably in the next 3 years to wondering if we might see i…

> The intelligence that will be available to the average technically literate individual will be frightening. That's not the scary part. The scary part is the intelligence at scale that could be available to the average employer . Lots of us like to LARP that we're capitalists, but very few of us are. There's zero ideological or cultural framework in place to prioritize the well being of the general population over t…

> AI, especially accelerating AI, is bad news for anyone who needs to work for a living. It's not going to lead to a Star Trek fantasy. It means an eventual phase change for the economy that consigns us (and most consumer product companies) to wither and fade away.

How would that work? If there are no consumers then why even bother producing? If the cost of labor and capital trends towards zero then the natural consequence is incredible deflation. If the producers refuse to lower their prices then they either don’t participate in the market (which also means their production is pointless) or ensure some other way that the consumers can buy their products.

Our society isn’t really geared for handling double digit deflation so something does need to change if we really are accelerating exponentially.

Re: S1: A $6 R1 competitor?

#286

Post by svara »

It just occurred to me that if you squint a little (just a little!) the S1 paper just provided the scientific explanation for why Twitter's short tweets mess you up and books are good for you.

Kidding, but not really. It's fascinating how we seem to be seeing a gradual convergence of machine learning and psychology.

Re: S1: A $6 R1 competitor?

#287

Post by raducu »

gessha wrote: post #204
teruakohatu wrote: post #193

Earlier quoted context omitted.

> still have no real comprehensive understanding how the models work. We do understand how they work, we just have not optimised their usage. For example someone who has a good general understanding of how an ICE or EV car works. Even if the user interface is very unfamiliar, they can figure out how to drive any car within a couple of minutes. But that does not mean they can race a car, drift a car or drive a car on…

Your example is somewhat inadequate. We _fundamentally_ don’t understand how deep learning systems works in the sense that they are more or less black boxes that we train and evaluate. Innovations in ML are a whole bunch of wizards with big stacks of money changing “Hmm” to “Wait” and seeing what happens. Would a different sampler help you? I dunno, try it. Would a smaller dataset help? I dunno, try it. Would trainin…

> _fundamentally_ don’t understand how deep learning systems works.

It's like saying we don't understand how quantum chromodynamics works. Very few people do, and it's the kind of knowledge not easily distilled for the masses in an easily digestible in a popsci way.

Look into how older CNNs work -- we have very good visual/accesible/popsci materials on how they work.

I'm sure we'll have that for LLM but it's not worth it to the people who can produce that kind of material to produce it now when the field is moving so rapidly, those people's time is much better used in improving the LLMs.

The kind of progress being made leads me to believe there absolutely ARE people who absolutely know how the LLMs work and they're not just a bunch of monkeys randomly throwing things at GPUs and seeing what sticks.

Re: S1: A $6 R1 competitor?

#288

Post by _w1tm »

palmotea wrote: post #271
lodovic wrote: post #268

Earlier quoted context omitted.

That's a fallacy. You can't have an advanced economy with most people sitting on the side. Money needs to keep flowing. If all that remains of the economy consists of a few datacenters talking to each other, how can the ruling class profit off that?

> You can't have an advanced economy with most people sitting on the side. If AI lives up to the hype, that will become possible. > If all that remains of the economy consists of a few datacenters talking to each other, how can the ruling class profit off that? I don't think it would be that. There'd also be power generation, manufacturing, mining, and construction, etc.; but all extremely automated. If you get to tr…

> It actually might kinda look like the "economy" of Starcraft: you gather resources, decide what to build with them, and order it all around according to your whim. There will be a handful of guys playing, and everyone else will be a NPC.

I guess if the “players” are sociopathic enough they might decide to just wipe out the NPCs. The possibility of someone like Putin or Musk becoming the sole member of the post-singularity humanity does make me pause.

Re: S1: A $6 R1 competitor?

#289

Post by tomaskafka »

mtrovo wrote: post #133

I found the discussion around inference scaling with the 'Wait' hack so surreal. The fact such an ingeniously simple method can impact performance makes me wonder how many low-hanging fruit we're still missing. So weird to think that improvements on a branch of computer science is boiling down to conjuring the right incantation words, how you even change your mindset to start thinking this way?

One thing is to realize that we as humans have a thinking steps (internal monologue) before we output the texts. When LLMs produce text, we expect this thinking process to happen as well, but it does not - they are 'idiots that babble the first thing that comes to their minds'.

The above 'hack' is one of many realizations of the above differences.

Re: S1: A $6 R1 competitor?

#290

Post by codeulike »

mtrovo wrote: post #133

I found the discussion around inference scaling with the 'Wait' hack so surreal. The fact such an ingeniously simple method can impact performance makes me wonder how many low-hanging fruit we're still missing. So weird to think that improvements on a branch of computer science is boiling down to conjuring the right incantation words, how you even change your mindset to start thinking this way?

Wait, so the trick is they reach into the context and basically switch '' with 'wait' and that makes it carry on thinking?
Post reply on HN