61–70 of 233 posts
Until LLMs have some 0% error humans will have to be in the loop (even if they only serve to take responsibility of the process).
The popular retelling of how Einstein created Special Relativity to "Resolve the contradictions of Michelson-Morly experiments" is very reductive to the history of the question. The epitome is the quote from the paper: > From the two postulates, Einstein derived the Lorentz trans- formation ... If Einstein derived them, who is "Lorentz"? The groundwork for Special Relativity was the study of electrodynamics and symme…
> If Einstein derived them, who is "Lorentz"?
You can (re-) derive a lot of existing stuff.
Earlier quoted context omitted.
Lots of things changed, GPT-2 is small (1.5e9) and is also a base model, so it is only doing next-token/autocomplete rather than prompt-response like even the first ChatGPT-3.5 was doing.
Just for the sake of clarity: all LLMs up to today are still only doing next-token/autocomplete. The training process got additional stages to shape the model weights, but standalone LLMs are still deployed essentially identically.
IMO, the mechanism isn't the important thing, the behaviour is. If you look at the step-by-step, we are also looking for the next word or motor action (and for whoever is about to suggest that we humans plan ahead, Transformer-based LLMs have been shown to also do this); as this is not a useful description of what it means to be a living brain, I'd say it's also not a useful description of what makes everything post-InstructGPT different from what came before.
Came for: "A computer once beat me at chess, but it was no match for me at kick boxing." TFA was actually about leaps of intuition, sadly. One of the experiments I've heard proposed around here is to somehow create an LLM from all text up to 1980 or 1990 and see if it can get back to making itself.
> One of the experiments I've heard proposed around here is to somehow create an LLM from all text up to 1980 or 1990 and see if it can get back to making itself. Could be, but preventing leakage from more modern stuff can be challenging. This was attempted with Victorian public domain content: https://www.estragon.news/mr-chatterbox-or-the-modern-promet... I can't find the citation right now, but I think people foun…
Earlier quoted context omitted.
Just for the sake of clarity: all LLMs up to today are still only doing next-token/autocomplete. The training process got additional stages to shape the model weights, but standalone LLMs are still deployed essentially identically.
If you gave GPT-2 a question and ended with a "?", it might answer, but also it might write several more questions in a similar category. IMO, the mechanism isn't the important thing, the behaviour is. If you look at the step-by-step, we are also looking for the next word or motor action (and for whoever is about to suggest that we humans plan ahead, Transformer-based LLMs have been shown to also do this); as this is…
Clearly LLMs cant do leaps of intuition since their "intuition" is locked after training ends. The only way a LLM can come up with new ideas if the "idea" appeared as a generalisation durring training or if it was achieved using reason in chain of thought.
Earlier quoted context omitted.
Moreover anyone glomming onto this paper for goal-post-shifting “AI can never” should: 1. Read the last sentence of the abstract, and 2. Reflect that frontier reasoning agents already increasingly integrate multimodal models.
The worst thing about the AI boom is how tech bros feel comfortable abusing the goalpost fallacy. So annoying.
I simply point out here, that fully accepting the paper’s premise, the paper’s conclusion isn’t limiting on frontier AI reasoning agents. The paper posits the necessity of multimodal world models and the limitations of LLMs. Frontier agents aren’t simply LLMs and do increasingly integrate increasingly capable multimodal models.
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
> When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
Earlier quoted context omitted.
> One of the experiments I've heard proposed around here is to somehow create an LLM from all text up to 1980 or 1990 and see if it can get back to making itself. Could be, but preventing leakage from more modern stuff can be challenging. This was attempted with Victorian public domain content: https://www.estragon.news/mr-chatterbox-or-the-modern-promet... I can't find the citation right now, but I think people foun…
Progress followed improvements in hardware, would you have to give access to modern hardware in the experiment for it to use? How much could it infer from it?
At a minimum, yes. IIRC, the sum total of all compute manufactured over history only reached the minimum needed to train an OK LMM in the mid 00s.
> How much could it infer from it?
Only way to find out is to try.