291–300 of 430 posts
Earlier quoted context omitted.
We know how the next token is selected, but not why doing that repeatedly brings all the capabilities it does. We really don't understand how the emergent behaviours emerge.
It feels less like a word prediction algorithm and more like a world model compression algorithm. Maybe we tried to create one and accidentaly created the other?
Earlier quoted context omitted.
I like the analogy of compression, in that a distilled model of an LLM is like a JPEG of a photo. Pretty good, maybe very good, but still lossy. The question I hear you raising seems to be along the lines of, can we use a new compression method to get better resolution (reproducibility of the original) in a much smaller size.
> in that a distilled model of an LLM is like a JPEG of a photo That's an interesting analogy, because I've always thought of the hidden states (and weights and biases) of an LLMs as a compressed version of the training data.
> In s1, when the LLM tries to stop thinking with " ", they force it to keep going by replacing it with "Wait". It’ll then begin to second guess and double check its answer. They do this to trim or extend thinking time (trimming is just abruptly inserting " ") I know some are really opposed to anthropomorphizing here, but this feels eerily similar to the way humans work, ie. if you just dedicate more time to analyzin…
What’s missing in that analogy is that humans tend to have a good hunch about when they have to think more and when they are “done”. LLMs seem to be missing a mechanism for that kind of awareness.
Earlier quoted context omitted.
I've noticed that R1 says "Wait," a lot in its reasoning. I wonder if there's something inherently special in that token.
I bet a token like "sh t!", "f* " or "damn!" would have the same or even stronger effect but the LLM creators would not like to have the users read them
Earlier quoted context omitted.
> You can't have an advanced economy with most people sitting on the side. If AI lives up to the hype, that will become possible. > If all that remains of the economy consists of a few datacenters talking to each other, how can the ruling class profit off that? I don't think it would be that. There'd also be power generation, manufacturing, mining, and construction, etc.; but all extremely automated. If you get to tr…
> It actually might kinda look like the "economy" of Starcraft: you gather resources, decide what to build with them, and order it all around according to your whim. There will be a handful of guys playing, and everyone else will be a NPC. I guess if the “players” are sociopathic enough they might decide to just wipe out the NPCs. The possibility of someone like Putin or Musk becoming the sole member of the post-sing…
This thing that people are calling “reasoning” is more like rendering to me really, or multi pass rendering. We’re just refining the render, there’s no reasoning involved.
For all the hype about thinking models, this feels much like compression in terms of information theory instead of a "takeoff" scenario. There are a finite amount of information stored in any large model, the models are really good at presenting the correct information back, and adding thinking blocks made the models even better at doing that. But there is a cap to that. Just like how you can compress a file by a lot…
As reasoning improves the models could start with a basic set of principles and build from there. Of course for facts grounded in reality RAG would still likely be the best, but maybe with enough “reasoning” a model could simulate an approximation of the universe well enough to get to an answer.
Earlier quoted context omitted.
I agree with you and I am scared. My problem is: if most people can't work, who is going to pay for the product/services created with IA? I get a lot of "IA will allow us to create SaaS in a weekend" and "IA will take engineers jobs", which I think they both may be true. But a lot of SaaS surive because engineers pay for them -- if engineer don't exist anymore, a lot of SaaS won't either. If you eat your potential cu…
> My problem is: if most people can't work, who is going to pay for the product/services created with IA? A lot of those will probably go under, too. I think a lot of people are in for a rude awakening. The only people our society and economy really values are the elite with ownership and control, and the people who get to eat and have comfort are those who provide things that are directly or indirectly valuable to t…
In case you’re not sure what S1 is, here is the original paper: https://arxiv.org/html/2501.19393v1
I found the discussion around inference scaling with the 'Wait' hack so surreal. The fact such an ingeniously simple method can impact performance makes me wonder how many low-hanging fruit we're still missing. So weird to think that improvements on a branch of computer science is boiling down to conjuring the right incantation words, how you even change your mindset to start thinking this way?
Wait, so the trick is they reach into the context and basically switch ' ' with 'wait' and that makes it carry on thinking?
>In s1, when the LLM tries to stop thinking with "", they force it to keep going by replacing it with "Wait".