Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on OpenAI's new o1 chain-of-thought models

simonwillison.net

Post reply on HN
Print view View original

101–110 of 659 posts

Re: Notes on OpenAI's new o1 chain-of-thought models

#101

Post by burtonator »

layer8 wrote: post #18

The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…

I'm honestly confused as to why it is doing this and why it thinks I'm right when I tell it that it is incorrect.

I've tried asking it factual information, and it asserts that it's incorrect but it will definitely hallucinate questions like the above.

You'd think the reasoning would nail that and most of the chain-of-thought systems I've worked on would have fixed this by asking it if the resulting answer was correct.

Re: Notes on OpenAI's new o1 chain-of-thought models

#102

Post by doe_eyes »

lukev wrote: post #85
fnordpiglet wrote: post #73

Earlier quoted context omitted.

One aspect that’s not achievable is they discuss hiding the chain of thought in its raw form because the chains are allowed to be unaligned. This allows the model to operate without any artifacts from alignment and apply them in the post processing, more or less. This requires effectively root and you would need the unaligned weights.

Ok but this presses on a latent question: what do we mean by alignment? Practically it's come to mean just sanitization... "don't say something nasty or embarrassing to users." But that doesn't apply here, the reasoning tokens are effectively just a debug log. If alignment means "conducting reasoning in alignment with human values", then misalignment in the reasoning phase could potentially be obfuscated and sanitize…

Alignment started as a fairly nifty idea, but you can't meaningfully test for it. We don't have the tools to understand the internals of an LLM.

So yes, it morphed into the second best thing, brand safety - "don't say racist / anti-vax stuff so that we don't get bad press or get in trouble with the regulators".

Re: Notes on OpenAI's new o1 chain-of-thought models

#103

Post by henry2023 »

>> which is larger 3.9, 3.10 or 3.11

To determine which number is larger among 3.9, 3.10, and 3.11, let's compare Standard Decimal Interpretation:

3.9 is simply 3.9. 3.10 is equivalent to 3.1 because trailing zeros after the decimal point do not add value in decimal numbers. 3.11 remains 3.11. Comparison:

3.9 vs. 3.1 vs. 3.11 3.11 is the largest number. Version Number Interpretation:

In version numbering, each segment separated by a dot is compared as an integer. 3.9 (major version 3, minor version 9) 3.10 (major version 3, minor version 10) 3.11 (major version 3, minor version 11) Comparison:

Minor versions: 11 > 10 > 9 3.11 is the largest version. Answer: 3.11 is the largest number among 3.9, 3.10, and 3.11.

...

So IMO level right?

Re: Notes on OpenAI's new o1 chain-of-thought models

#104

Post by kristianp »

I tried a problem I was looking at recently, to refactor a small rust crate to use one datatype instead of an enum, to help me understand the code better. I found o1-mini made a decent attempt, but couldn't provide error free code. o1-preview was able to provide code that compiled and passed all but the test that is expected to fail, given the change I asked it to make.

This is the prompt I gave:

simplify this rust library by removing the different sized enums and only using the U8 size. For example MasksByByte is an enum, change it to be an alias for the U8 datatype. Also the u256 datatype isn't required, we only want U8, so remove all references to U256 as well.

The original crate is trie-hard [1][2] and I forked it and put the models attempts in the fork [3]. I also quickly wrote it up at [4]

[1] https://blog.cloudflare.com/pingora-saving-compute-1-percent...

[2] https://github.com/cloudflare/trie-hard

[3] https://github.com/kpm/trie-hard-simple/tree/main/attempts

[4] https://blog.reyem.dev/post/refactoring_rust_with_chatgpt-o1...

Re: Notes on OpenAI's new o1 chain-of-thought models

#105

Post by waveBidder »

elicksaur wrote: post #71
helmsb wrote: post #33

I did a few tests and asked it some legal questions. 4o gave me the correct answer immediately. o1 preview gave a much more in depth but completely wrong answer. It took 5 follow ups to get it to recognize that it hallucinated a non-existent law

This is interesting since they claim it does well on STEM questions, which I’d assume would be a similar level of reasoning complexity for a human.

A difficult to guess fraction of all of these results are training to the test in various forms

Re: Notes on OpenAI's new o1 chain-of-thought models

#106

Post by MPSimmons »

layer8 wrote: post #18

The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…

The failure is in how you're using it. I don't mean this as a personal attack, but more to shed light on what's happening.

A lot of people use LLMs as a search engine. It makes sense - it's basically a lossy compressed database of everything its ever read, and it generates output that is statistically likely - varying degrees of likeliness depending on the temperature, as well as how many times the particular weights your prompt ends up activating.

The magic of LLMs, especially one like this that supposedly has advanced reasoning, isn't the existing knowledge in its weights. The magic is that _it knows english_. It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output. It's not _just_ an output engine. It's an engine that outputs.

Asking it about nuanced details in the corpus of data it has read won't give you good output unless it read a bunch of it.

On the other hand, if you were to paste the entire documentation set to a tool it has never seen and ask it to use the tool in a way to accomplish your goals, THEN this model would be likely to produce useful output, despite the fact that it had never encountered the tool or its documentation before.

Don't treat it as a database. Treat it as a naive but intelligent intern. Provide it data, give it a task, and let it surprise you with its output.

Re: Notes on OpenAI's new o1 chain-of-thought models

#107

Post by cdelsolar »

elicksaur wrote: post #62
salomonk_mur wrote: post #59

Earlier quoted context omitted.

The innovation lies in making the whole loop available to an end user immediately, without them being a programmer. My grandma can build games using ChatGPT now.

What games have people made with ChatGPT? Do you have an example of a live, deployed game?

Yes, a gazillion of them. Someone in a scrabble Facebook group made this entirely with ChatGPT: https://aboocher.github.io/scrabble/ingpractice.html

Re: Notes on OpenAI's new o1 chain-of-thought models

#108

Post by rcarr »

layer8 wrote: post #18

The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…

Has anyone tried asking it to generate the libraries/functions that it's hallucinating and seeing if it can do so correctly? And then seeing if it can continue solving the original problem with the new libraries? It'd be absolutely fascinating if it turns out it could do this.

Re: Notes on OpenAI's new o1 chain-of-thought models

#109

Post by throwaway314155 »

henry2023 wrote: post #103

>> which is larger 3.9, 3.10 or 3.11 To determine which number is larger among 3.9, 3.10, and 3.11, let's compare Standard Decimal Interpretation: 3.9 is simply 3.9. 3.10 is equivalent to 3.1 because trailing zeros after the decimal point do not add value in decimal numbers. 3.11 remains 3.11. Comparison: 3.9 vs. 3.1 vs. 3.11 3.11 is the largest number. Version Number Interpretation: In version numbering, each segmen…

> So IMO level right?

What?

Re: Notes on OpenAI's new o1 chain-of-thought models

#110

Post by pedrosorio »

MPSimmons wrote: post #106
layer8 wrote: post #18

The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…

The failure is in how you're using it. I don't mean this as a personal attack, but more to shed light on what's happening. A lot of people use LLMs as a search engine. It makes sense - it's basically a lossy compressed database of everything its ever read, and it generates output that is statistically likely - varying degrees of likeliness depending on the temperature, as well as how many times the particular weights…

> It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output

This is not an apt description of the system that insists the doctor is the mother of the boy involved in a car accident when elementary understanding of English and very little logic show that answer to be obviously wrong.

https://x.com/colin_fraser/status/1834336440819614036

Post reply on HN