Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on OpenAI's new o1 chain-of-thought models

simonwillison.net

Post reply on HN
Print view View original

641–650 of 659 posts

Re: Notes on OpenAI's new o1 chain-of-thought models

#641

Post by KoolKat23 »

bnralt wrote: post #429
KoolKat23 wrote: post #363

Earlier quoted context omitted.

I'm noticing a strange common theme in all these riddles, it's being asked and getting wrong. They're all badly worded questions. The model knows something is up and reads into it too much. In this case it's tautology, you would usually say "a mother and her son...". I think it may answer correctly if you start off asking "Please solve the below riddle:" There was another example yesterday which it solved correctly a…

> They're all badly worded questions. The model knows something is up and reads into it too much. The model knows something is up and reads into it too much. In this case it's tautology, you would usually say "a mother and her son...". How is "a woman and her son" badly worded? The meaning is clear and blatently obvious to any English speaker.

Go read the whole riddle, add the rest of it and you'll see it's contrived, hence it's a riddle even for humans. The model in it's thinking (which you can read) places undue influence on certain anomalous factors. In practice, a person would say this way more eloquently than the riddle.

Re: Notes on OpenAI's new o1 chain-of-thought models

#642

Post by KoolKat23 »

grey-area wrote: post #490
Jensson wrote: post #472

Earlier quoted context omitted.

> especially when evaluated on the first snap answer The whole point of o1 is that it wasn't "the first snap answer", it wrote half a page internally before giving the same wrong answer.

Is that really its internal 'chain of thought' or is it a post-hoc justification generated afterward? Do LLMs have a chain of thought like this at all or are they just convincing at mimicking what a human might say if asked for a justification for an opinion?

Its slightly more strange than this as both are true. It's already baked in the model but chain of thought does improve reasoning, you only have to look at maths problems. A short guess would be wrong but it would get it correct if asked to break it down and reason (harder to see nowadays as it has access to calculators).

Re: Notes on OpenAI's new o1 chain-of-thought models

#643

Post by troupo »

KoolKat23 wrote: post #640
troupo wrote: post #629

Earlier quoted context omitted.

> Perfection is the enemy of good. Where exactly did I write anything about perfection? For me "AIs" are incapable of producing working code: https://news.ycombinator.com/item?id=41534233

You said you have to babysit each line of code, I mean this is simply untrue, if it works there's no need to babysit, the only reason you'd need to babysit every single line is if you're looking for perfection or it's something very obscure or unheard of. Your example is perhaps valid, but there are other examples where it does work as I mentioned. I think it may be imprecise prompting, too general or with too little…

> You said you have to babysit each line of code, I mean this is simply untrue, if it works there's no need to babysit

No. It either doesn't work, or works incorrectly, or the code is incomplete despite requirements etc.

> Your example is perhaps valid, but there are other examples where it does work as I mentioned.

It's funny how I'm supposed to assume your examples are the truth, and nothing but the truth, but my examples are "untrue, you're a perfectionist, and perhaps you're right"

> the more detail and more technical you speak the better

As I literally wrote in the comment you're so dismissive of: "As for "using LLMs wrong", using them "right" is literally babysitting their output and spending a lot of time trying to reverse-engineer their behavior with increasingly inane prompts."

> assume it's a very precise expert.

If it was an expert, as you claim it to be, it would not need extremely detailed prompting. As it is, it's a willing but clumsy junior.

To the point that it would rewrite the code I fixed with invalid code when asked to fix an unrelated mistake.

> Good prompting and verifying output

How is it you repeat everything I say, and somehow assume I'm wrong and my examples are invalid?

Re: Notes on OpenAI's new o1 chain-of-thought models

#644

Post by idiotsecant »

HarHarVeryFunny wrote: post #587
throwaway4aday wrote: post #575

Earlier quoted context omitted.

I would think they'd be more likely to form the language centre of a composite AGI brain. If you read through the known functions of the various areas involved in language[0] they seem to map quite well to the capabilities of transformer based LLMs especially the multi-modal ones. [0] https://en.wikipedia.org/wiki/Language_center

It's not obvious that an LLM - a pre-trained/frozen chunk of predictive statistics - would be amenable to being used as an integral part of an AGI that would necessarily be using a different incremental learning algorithm. Would the transformer architecture be compatible with the needs of an incremental learning system? It's missing the top down feedback paths (finessed by SGD training) needed to implement prediction…

Its entirely possible to have an AGI language model that is periodically retrained as slang, vernacular, and semantic embeddings shift in their meaning. I have little doubt that something very much like an LLM (a machine that turns high dimensional intent into words) will form an AGIs 'language center' at some point.

Re: Notes on OpenAI's new o1 chain-of-thought models

#645

Post by KoolKat23 »

troupo wrote: post #643
KoolKat23 wrote: post #640

Earlier quoted context omitted.

You said you have to babysit each line of code, I mean this is simply untrue, if it works there's no need to babysit, the only reason you'd need to babysit every single line is if you're looking for perfection or it's something very obscure or unheard of. Your example is perhaps valid, but there are other examples where it does work as I mentioned. I think it may be imprecise prompting, too general or with too little…

> You said you have to babysit each line of code, I mean this is simply untrue, if it works there's no need to babysit No. It either doesn't work, or works incorrectly, or the code is incomplete despite requirements etc. > Your example is perhaps valid, but there are other examples where it does work as I mentioned. It's funny how I'm supposed to assume your examples are the truth, and nothing but the truth, but my e…

I did not say your examples are untrue, no need to be so defensive. Believe what you wish but my example is true and works. A willing but clumsy junior benefits tremendously from a well scoped task.

Re: Notes on OpenAI's new o1 chain-of-thought models

#646

Post by HarHarVeryFunny »

idiotsecant wrote: post #644
HarHarVeryFunny wrote: post #587

Earlier quoted context omitted.

It's not obvious that an LLM - a pre-trained/frozen chunk of predictive statistics - would be amenable to being used as an integral part of an AGI that would necessarily be using a different incremental learning algorithm. Would the transformer architecture be compatible with the needs of an incremental learning system? It's missing the top down feedback paths (finessed by SGD training) needed to implement prediction…

Its entirely possible to have an AGI language model that is periodically retrained as slang, vernacular, and semantic embeddings shift in their meaning. I have little doubt that something very much like an LLM (a machine that turns high dimensional intent into words) will form an AGIs 'language center' at some point.

Yes, an LLM can be periodically retrained, which is what is being done today, but a human level AGI needs to be able to learn continuously.

If we're trying something new and make a mistake, then we need to seamlessly learn from the mistake and continue - explore the problem and learn from successes and failures. It wouldn't be much use if your "AGI" intern stopped at it's first mistake and said "I'll be back in 6 months after I've been retrained not to make THAT mistake".

Re: Notes on OpenAI's new o1 chain-of-thought models

#647

Post by dartos »

KoolKat23 wrote: post #639
dartos wrote: post #637

Earlier quoted context omitted.

You're saying that because two different objects can to solve the same problem, they must work the same way. An abacus and a calculator were both made to solve relativly simple math problems, so they must work in the same way, right? And apple and an orange are ways to store sugar for plants, so they must be the same thing, right? No. That's not how any of this works. An abacus and a calculator are two different tool…

No I'm not saying they must work the same way. I'm saying it's evidence there is a more efficient way as they both solve the same problem and one is more efficient (in truth both are more efficient in different areas). At an abstract level they can be doing the same thing. What does a simulator do? Think a little further yes, currently it's a brand new model each time but why will it be this way forever? Its an engin…

So you’re just abstracting everything to the point where everything is a “something solver” and if two things can solve the same something, one must be a better version of the other?

Abstracting everything to the point of meaninglessness isn’t a worthwhile exercise.

Re: Notes on OpenAI's new o1 chain-of-thought models

#648

Post by grey-area »

TeMPOraL wrote: post #635
grey-area wrote: post #615

Earlier quoted context omitted.

I'm curious what you think is happening here as your answer seems to imply it is thinking (and indeed rushing to an answer somehow). Do you think the generative AI has agency or a thought process? It doesn't seem to have anything approaching that to me, nor does it answer quickly. It seems to be more like a weighing machine based on past tokens encountered together, so this is exactly the kind of answer we'd expect o…

My thinking is that LLMs are very similar, perhaps structurally the same, as a piece of human brain that does the "inner voice" thing. The boundary between the subconscious and conscious, that generates words and phrases and narratives pretty much like "feels best" autocomplete[0] - bits that other parts of your mind evaluate and discard, or circle back, because if you were just to say or type directly what your inne…

Interesting perspective, thanks. I can’t help but feel they are still missing a major part of cognition though which is having a stable model of the world.

Re: Notes on OpenAI's new o1 chain-of-thought models

#649

Post by codelion »

alchemist1e9 wrote: post #390
codelion wrote: post #89

Earlier quoted context omitted.

I have also spent some time on 2) and implemented several approaches in this open source optimising llm proxy - https://github.com/codelion/optillm In my experience it does work quite well, but we probably need different techniques for different tasks.

Maybe 1 is actually hat you just suggested - an RL approach to select the strategy for 2. Thank you for implementing optillm and working out all the various strategy options, it’s a really neat reference for understanding this space. One item I’m very curious about is how do they get a score for use in the RL? in well defined games it’s easy to understand but in this LLM output context how does one rate the output re…

That’s the hardest part, figuring out the reward. For generic tasks it is not easy, in my implementation in optillm I am using the llm itself to generate a score based on the mcts trajectory. But that is not as good as having a reward that is well defined say for a coding or logic problem. May be they trained a better reward model.

Re: Notes on OpenAI's new o1 chain-of-thought models

#650

Post by anon291 »

roywiggins wrote: post #436
anon291 wrote: post #111

Earlier quoted context omitted.

Keep in mind that the system always chooses randomly so there is always a possibility it commits to the wrong output. I don't know why openAi won't allow determinism but it doesn't, even with temperature set to zero

Determinism only helps if you always ask the question with exactly the same words. There's no guarantee a slightly rephrased version will give the same answer, so a certain amount of unpredictability is unavoidable anyway. With a deterministic LLM you might find one phrasing that always gets it right and a dozen basically indistinguishable ones that always get it wrong.

My program always asks the same question yes.
Post reply on HN