S1: A $6 R1 competitor?

Post by ForHackernews » Thu, Feb 06, 2025, 1:22 PM UTC

zoogeny wrote: post #196

pona-a wrote: post #53

If chain of thought acts as a scratch buffer by providing the model more temporary "layers" to process the text, I wonder if making this buffer a separate context with its own separate FNN and attention would make sense; in essence, there's a macroprocess of "reasoning" that takes unbounded time to complete, and then there's a microprocess of describing this incomprehensible stream of embedding vectors in natural lan…

I've had an idea since I was a kid which I can share. I was contemplating AI and consciousness generally, probably around the time I read "The Minds I". I reflected on the pop-psychology idea of consciousness and subconsciousness. I thought of each as an independent stream of tokens, like stream of consciousness poetry. But along the stream there were joining points between these two streams, points where the conscio…

Post by ijk » Thu, Feb 06, 2025, 1:23 PM UTC

gessha wrote: post #309

brookst wrote: post #262

Earlier quoted context omitted.

Isn't that just scale? Even small LLMs have more parts than any car. LLMs are more analogous to economics, psychology, politics -- it is possible there's a core science with explicability, but the systems are so complex that even defining the question is hard.

You can make a bigger ICE engine (like a container ship engine) and still understand how the whole thing works. Maybe there’s more parts moving but it still has the structure of an ICE engine. With neural networks big or small, we got no clue what’s going on. You can observe the whole system, from the weights and biases, to the activations, gradients, etc and still get nothing. On the other hand, one of the reasons w…

Post by bloomingkales » Thu, Feb 06, 2025, 1:29 PM UTC

gessha wrote: post #308

raducu wrote: post #287

Earlier quoted context omitted.

> _fundamentally_ don’t understand how deep learning systems works. It's like saying we don't understand how quantum chromodynamics works. Very few people do, and it's the kind of knowledge not easily distilled for the masses in an easily digestible in a popsci way. Look into how older CNNs work -- we have very good visual/accesible/popsci materials on how they work. I'm sure we'll have that for LLM but it's not wort…

As a person who has trained a number of computer vision deep networks, I can tell you that we have some cool-looking visualizations on how lower layers work but no idea how later layers work. The intuition is built over training numerous networks and trying different hyperparameters, data shuffling, activations, etc. it’s absolutely brutal over here. If the theory was there, people like Karpathy who have great teache…

Post by eru » Thu, Feb 06, 2025, 1:34 PM UTC

advael wrote: post #277

I'm strictly speaking never going to think of model distillation as "stealing." It goes against the spirit of scientific research, and besides every tech company has lost my permission to define what I think of as theft forever

Post by timschmidt » Thu, Feb 06, 2025, 1:42 PM UTC

umeshunni wrote: post #269

MR4D wrote: post #259

Earlier quoted context omitted.

I like the analogy of compression, in that a distilled model of an LLM is like a JPEG of a photo. Pretty good, maybe very good, but still lossy. The question I hear you raising seems to be along the lines of, can we use a new compression method to get better resolution (reproducibility of the original) in a much smaller size.

> in that a distilled model of an LLM is like a JPEG of a photo That's an interesting analogy, because I've always thought of the hidden states (and weights and biases) of an LLMs as a compressed version of the training data.