Live data from Hacker News

Notes on OpenAI's new o1 chain-of-thought models

simonwillison.net

71–80 of 659 posts

Re: Notes on OpenAI's new o1 chain-of-thought models

#71
post #33

I did a few tests and asked it some legal questions. 4o gave me the correct answer immediately. o1 preview gave a much more in depth but completely wrong answer. It took 5 follow ups to get it to recognize that it hallucinated a non-existent law

This is interesting since they claim it does well on STEM questions, which I’d assume would be a similar level of reasoning complexity for a human.

Re: Notes on OpenAI's new o1 chain-of-thought models

#72

I've just wasted a few rounds of my weekly o1 ammo by feeding it hard problems I have been working on over the last couple days and for which GPT-4o had failed spectacularly. I suppose I'm to blame for raising my own expectations after the latest PR, but I was pretty disappointed when the answers weren't any better from what I got with the old model. TL;DR It felt less like a new model and way more like one of those…

Can you give an example of one of these problems for context?

Re: Notes on OpenAI's new o1 chain-of-thought models

#73
post #43

It's interesting to note that there's really two things going on here: 1. A LLM (probably a finetuned GPT-4o) trained specifically to read and emit good chain-of-thought prompts. 2. Runtime code that iteratively re-prompts the model with the chain of thought so far. This sounds like it includes loops, branches and backtracking. This is not "the model", it's regular code invoking the model. Interesting that OpenAI is…

One aspect that’s not achievable is they discuss hiding the chain of thought in its raw form because the chains are allowed to be unaligned. This allows the model to operate without any artifacts from alignment and apply them in the post processing, more or less. This requires effectively root and you would need the unaligned weights.

Re: Notes on OpenAI's new o1 chain-of-thought models

#74
post #57

Earlier quoted context omitted.

No, ChatGPT is way cooler than that. It's already read every line of kernel code ever written. I start with a subsystem: the device mapper is a good recent example. I ask things like "explain the linux device mapper. if it was a class in an object-oriented language, what would its interface look like?" and "give me dm_target as a python class". I get stuff like: def linear_ctr(target, argc, argv): print("Constructor…

That's awesome. Have you tried asking it to convert Python (psuedo-ish) code back into C that interfaces with the kernel?

No, but only because I have no use for it. I wouldn't be surprised if it did a fine job! I'd be remiss if I didn't note that it's way better at doing this for the Linux kernel than with codebases like Zookeeper and Kubernetes (though: maybe o1 makes this better, who knows?).

I do feel like someone who skipped like 8 iPhone models (cross-referencing, EIEIO, lsp-mode, code explorers, tree-sitter) and just got an iPhone 16. Like, nothing that came before this for code comprehension really matters all that much?

Re: Notes on OpenAI's new o1 chain-of-thought models

#75
post #51

Earlier quoted context omitted.

I honestly can’t believe this is the hyped up “strawberry” everyone was claiming is pretty much AGI. Senior employees leaving due to its powers being so extreme I’m in the “probabilistic token generators aren’t intelligence” camp so I don’t actually believe in AGI, but I’ll be honest the never ending rumors / chatter almost got to me Remember, this is the model some media outlet reported recently that is so powerful…

I mean, considering how many tokens their example prompt consumed, I wouldn't be surprised if it costs ~$2k/month/user to run

[deleted]

Re: Notes on OpenAI's new o1 chain-of-thought models

#76
post #65

Earlier quoted context omitted.

The innovation lies in making the whole loop available to an end user immediately, without them being a programmer. My grandma can build games using ChatGPT now.

No she can't, comments likes yours are just made up nonsense that AI hype-mans and investors somehow convinced us are a fair opinions to have.

While AI is overhyped by some people, the parent's statement is not only true but was true long before o1 was released.

Re: Notes on OpenAI's new o1 chain-of-thought models

#77
post #65

Earlier quoted context omitted.

The innovation lies in making the whole loop available to an end user immediately, without them being a programmer. My grandma can build games using ChatGPT now.

No she can't, comments likes yours are just made up nonsense that AI hype-mans and investors somehow convinced us are a fair opinions to have.

Check out replit agents, they can make games and apps autonomously now

Re: Notes on OpenAI's new o1 chain-of-thought models

#78
post #47

Earlier quoted context omitted.

I dunno man. I just spent a couple hours trying to get it to write functioning code to read from my RTSP stream, detect if my kid is playing piano, and send the result to HomeAssistant. It did not succeed.

How many hours without it?

Not the OP, but in my experience LLMs fail in ways that indicate they will never solve the problem.

Stuck in loops, correct their mistakes with worse mistakes, hallucinating things that don’t exist and being unable to correct.

Working on my own, I have the confidence that I know I can make incremental forward progress on a problem. That’s much preferable.

Re: Notes on OpenAI's new o1 chain-of-thought models

#79
Please please please stop saying thought. This has nothing to do with the word thought. When we say the word thought it means something. Please don’t use the same word for whatever AI is doing and trivialize the word. Invent a new word if needed but for Pete’s sake be accurate and truthful.

Re: Notes on OpenAI's new o1 chain-of-thought models

#80
post #65

Earlier quoted context omitted.

No she can't, comments likes yours are just made up nonsense that AI hype-mans and investors somehow convinced us are a fair opinions to have.

Check out replit agents, they can make games and apps autonomously now

Do you know of an example game I can play right now?
Post reply on HN