Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't know how you get here from “predict the next word”

grumpy-economist.com

Post reply on HN
Print view View original

121–130 of 275 posts

Re: I don't know how you get here from “predict the next word”

#121

Post by [deleted] »

measurablefunc wrote: post #55
brookst wrote: post #44

Earlier quoted context omitted.

Not the person you asked, and “novel” is a minefield. What’s the last novel anything , in the sense you can’t trace a precursor or reference? But.. I recently had a LLM suggest an approach to negative mold-making that was novel to me. Long story, but basically isolating the gross geometry and using NURBS booleans for that, plus mesh addition/subtraction for details. I’m sure there’s prior art out there, but that’s tr…

I don't know, that's why I asked b/c I always see a lot of empty platitudes when it comes to LLM praise so I'm curious to see if people can actually back up their claims. I haven't done any 3D modeling so I'll take your word for it but I can tell you that I am working on a very simple interpreter & bytecode compiler for a subset of Erlang & I have yet to see anything novel or even useful from any of the coding assist…

[deleted]

Re: I don't know how you get here from “predict the next word”

#122

Post by grey-area »

steve1977 wrote: post #103
grey-area wrote: post #70

Earlier quoted context omitted.

Given their failure on novel logic problems, generation of meaningless text, tendency to do things like delete tests and incompetence at simple mathematics, it seems very unlikely they have built any sort of world model. It’s remarkable how competent they are given the way they work. Predict the next word is a terrible summary of what these machines do though, they certainly do more than that, but there are significa…

> Predict the next word is a terrible summary of what these machines do though, they certainly do more than that What would that be?

They generate text based on quite a large context, including hidden prompts we don’t see and their weights are distorted heavily by training. So I think there’s a lot more than a simple probability of word x coming next. That makes ‘predict next word’ a reductive summary IMO.

I do not personally feel it resembles thinking or reasoning though and really object to that framing because it is misleading many people.

Re: I don't know how you get here from “predict the next word”

#123

Post by kmaitreys »

pushedx wrote: post #71
measurablefunc wrote: post #55

Earlier quoted context omitted.

I don't know, that's why I asked b/c I always see a lot of empty platitudes when it comes to LLM praise so I'm curious to see if people can actually back up their claims. I haven't done any 3D modeling so I'll take your word for it but I can tell you that I am working on a very simple interpreter & bytecode compiler for a subset of Erlang & I have yet to see anything novel or even useful from any of the coding assist…

Which agents are you using, and are you using them in an agent mode (Codex, Claude Code etc.)? The difference in quality of output between Claude Sonnet and Claude Opus is around an order of magnitude. The results that you can get from agent mode vs using a chat bot are around two orders of magnitude.

Can you clarify a bit more about the this two orders of magnitude? In what context? Sure, they have "agency" and can do more than outputting text, but I would like see a proper example of this claim.

Re: I don't know how you get here from “predict the next word”

#124

Post by kennyloginz »

brookst wrote: post #44
measurablefunc wrote: post #8

Earlier quoted context omitted.

What's the latest novel insight you have encountered?

Not the person you asked, and “novel” is a minefield. What’s the last novel anything , in the sense you can’t trace a precursor or reference? But.. I recently had a LLM suggest an approach to negative mold-making that was novel to me. Long story, but basically isolating the gross geometry and using NURBS booleans for that, plus mesh addition/subtraction for details. I’m sure there’s prior art out there, but that’s tr…

There is prior art, so it’s not novel.

Re: I don't know how you get here from “predict the next word”

#125

Post by steve1977 »

basch wrote: post #117
steve1977 wrote: post #115

Earlier quoted context omitted.

Thanks for the link, I will read it. But keep in mind that Google wants to sell us something.

A the time, it was a free language translation tool. You weren't paying for transformers in 2017.

True, but that doesn't mean that Google did not already have intentions to monetize it if possible.

Re: I don't know how you get here from “predict the next word”

#126

Post by dnautics »

mzhaase wrote: post #100
ChaitanyaSai wrote: post #43

The whole next word thing is interesting isn't it. I like to see it with Dennett's "Competence and comprehension" lens. You can predict the next word competently with shallow understanding. But you could also do it well with understanding or comprehension of the full picture. A mental model that allows you to predict better. Are the AIs stumbling into these mental models? Seems like it. However, because these are suc…

It always occurred to me that LLMs may be like the language center of the brain. And there should be a "whole damn rest of the brain" behind it to steer it. LLMs miss very important concepts, like the concept of a fact. There is no "true", just consensus text on the internet given a certain context. Like that study recently where LLMs gave wrong info if there was the biography of a poor person in the context.

thats unlikely. but they are awfully lot like turing machines (k/v cache ~ turing tape) so their architecture is strongly predisposed to be able to find any algorithm, possibly including reasoning

Re: I don't know how you get here from “predict the next word”

#127

Post by Kim_Bruning »

measurablefunc wrote: post #96
pushedx wrote: post #71

Earlier quoted context omitted.

Which agents are you using, and are you using them in an agent mode (Codex, Claude Code etc.)? The difference in quality of output between Claude Sonnet and Claude Opus is around an order of magnitude. The results that you can get from agent mode vs using a chat bot are around two orders of magnitude.

The workflow is not the issue. You are welcome to try the same challenge yourself if you want. Extra test cases ( https://drive.proton.me/urls/6Z6557R2WG#n83c6DP6mDfc ) & specification ( https://claude.ai/public/artifacts/5581b499-a471-4d58-8e05-1... ). I know enough about compilers, bytecode VMs, parsers, & interpreters to know that this is well within the capabilities of any reasonably good software engineer but th…

Possibly a dumb question: but are you running this in claude code, or an ide, or basically what are you using to allow for iteration?

Re: I don't know how you get here from “predict the next word”

#128

Post by qsera »

basch wrote: post #120
qsera wrote: post #118

Earlier quoted context omitted.

Yea, semantics is important. It is not "understanding" any more than a microphone+ADC is hearing.

agree to disagree. encoding a meaning is understanding. I cited a source using the word in the same way.

>agree to disagree.

Yea

>encoding a meaning is understanding.

encoding a meaning is encoding. Nothing more!

Re: I don't know how you get here from “predict the next word”

#130

Post by pushedx »

measurablefunc wrote: post #96
pushedx wrote: post #71

Earlier quoted context omitted.

Which agents are you using, and are you using them in an agent mode (Codex, Claude Code etc.)? The difference in quality of output between Claude Sonnet and Claude Opus is around an order of magnitude. The results that you can get from agent mode vs using a chat bot are around two orders of magnitude.

The workflow is not the issue. You are welcome to try the same challenge yourself if you want. Extra test cases ( https://drive.proton.me/urls/6Z6557R2WG#n83c6DP6mDfc ) & specification ( https://claude.ai/public/artifacts/5581b499-a471-4d58-8e05-1... ). I know enough about compilers, bytecode VMs, parsers, & interpreters to know that this is well within the capabilities of any reasonably good software engineer but th…

sorry, needed to edit this comment to ask the same question as the sibling:

have you run these models in an agent mode that allows for executing the tests, the agent views the output, and iterates on its own for a while? up to an hour or so?

you will get vastly different output if you ask the agent to write 200 of its own test cases, and then have it iterate from there

Post reply on HN