Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on OpenAI's new o1 chain-of-thought models

simonwillison.net

Post reply on HN
Print view View original

391–400 of 659 posts

Re: Notes on OpenAI's new o1 chain-of-thought models

#391

Post by idiotsecant »

berniedurfee wrote: post #381
williamdclt wrote: post #164

Earlier quoted context omitted.

> Treat it as a naive but intelligent intern That’s the problem: it’s a _terrible_ intern. A good intern will ask clarifying questions, tell me “I don’t know” or “I’m not sure I did it right”. LLMs do none of that, they will take whatever you ask and give a reasonable-sounding output that might be anything between brilliant and nonsense. With an intern, I don’t need to measure how good my prompting is, we’ll usually…

I’m starting to think this is an unsolvable problem with LLMs. The very act of “reasoning” requires one to know that they don’t know something. LLMs are giant word Plinko machines. A million monkeys on a million typewriters. LLMs are not interns. LLMs are assumption machines. None of the million monkeys or the collective million monkeys are “reasoning” or are capable of knowing. LLMs are a neat parlor trick and are s…

I think LLMs are definitely on the path to AGI in the same way that the ball bearing was on the path to the internal combustion engine. I think its quite likely that LLMs will perform important functions within the system of an eventual AGI.

Re: Notes on OpenAI's new o1 chain-of-thought models

#392

Post by TeMPOraL »

KoolKat23 wrote: post #363
pedrosorio wrote: post #110

Earlier quoted context omitted.

> It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output This is not an apt description of the system that insists the doctor is the mother of the boy involved in a car accident when elementary understanding of English and very little logic show that answer to be obviously wrong. https://x.com/colin_fraser/status/183…

I'm noticing a strange common theme in all these riddles, it's being asked and getting wrong. They're all badly worded questions. The model knows something is up and reads into it too much. In this case it's tautology, you would usually say "a mother and her son...". I think it may answer correctly if you start off asking "Please solve the below riddle:" There was another example yesterday which it solved correctly a…

Yup. The models fail on gotcha questions asked without warning, especially when evaluated on the first snap answer. Much like approximately all humans.

Re: Notes on OpenAI's new o1 chain-of-thought models

#393

Post by KoolKat23 »

troupo wrote: post #188
layer8 wrote: post #23

Earlier quoted context omitted.

My experience is likely colored by the fact that I tend to turn to LLMs for problems I have trouble solving by myself. I typically don't use them for the low-hanging fruits. That's the frustrating thing. LLMs don't materially reduce the set of problems where I'm running against a wall or have trouble finding information.

> That's the frustrating thing. LLMs don't materially reduce the set of problems where I'm running against a wall or have trouble finding information. As you step outside regular Stack Overflow questions for top-3 languages, you run into limitations of these predictive models. There's no "reasoning" behind them. They are still, largely, bullshit machines.

you're both on the wrong wavelength. No one has claimed it is better than an expert human yet. Be glad, for now your jobs are safe, why not use it as a tool to boost your productivity, yes, even though you'll get proportionally less use than others in other perhaps less "expert" jobs.

Re: Notes on OpenAI's new o1 chain-of-thought models

#394

Post by bartread »

ryoshu wrote: post #385
williamdclt wrote: post #164

Earlier quoted context omitted.

> Treat it as a naive but intelligent intern That’s the problem: it’s a _terrible_ intern. A good intern will ask clarifying questions, tell me “I don’t know” or “I’m not sure I did it right”. LLMs do none of that, they will take whatever you ask and give a reasonable-sounding output that might be anything between brilliant and nonsense. With an intern, I don’t need to measure how good my prompting is, we’ll usually…

1000% this. LLMs can't say "I don't know" because they don't actually think. I can coach a junior to get better. LLMs will just act like they know what they are doing and give the wrong results to people who aren't practitioners. Good on OAI calling their model Strawberry because of Internet trolls. Reactive vs proactive.

I get a lot of value out of ChatGPT but I also, fairly frequently, run into issues here. The real danger zones are areas that lie at or just beyond the edges of my own knowledge in a particular area.

I'd say that most of my work use of ChatGPT does in fact save me time but, every so often, ChatGPT can still bullshit convincingly enough to waste an hour or two for me.

The balance is still in its favour, but you have to keep your wits about you when using it.

Re: Notes on OpenAI's new o1 chain-of-thought models

#395

Post by dartos »

falcor84 wrote: post #305
deisteve wrote: post #58

Earlier quoted context omitted.

this is why i became skeptical of openai's claims if they shared the COT the grift wont work its just RL

I can't help but feel that saying "it's just RL" is like someone at the start of the 20th century saying "it's just electricity", as if understanding the underlying mechanism is the same as understanding the applications it can enable.

Tbf RL is pretty incredible.

I trained a model to play a novel video game using only screenshots and a score using RL and I discovered how not to lose

Re: Notes on OpenAI's new o1 chain-of-thought models

#396

Post by mewpmewp2 »

lucubratory wrote: post #182
hmottestad wrote: post #161

Earlier quoted context omitted.

Reminds me of a trick question about Schrödinger's cat. “I’ve put a dead cat in a box with a poison and an isotope that will trigger the poison at a random point in time. Right now, is the cat dead or alive?” The answer is that the cat is dead, because it was dead to begin with. Understanding this doesn’t mean that you are good at deductive reasoning. It just means that I didn’t manage to trick you. Same goes for an…

Yeah, I think what a lot of people miss about these sort of gotchas are that most of them were invented explicitly to gotcha humans, who regularly get got by them. This is not a failure mode unique to LLMs.

If there is attention mechanism then maybe that is what is fault, because if it is a common riddle attention mechanism only notices that it is a common riddle, not that there is a gotcha planted in. Because when I read the sentence myself, I did not immediately notice that the cat that was put in there was actually dead when it was put there, because I pattern matched this to a known problem, I did not think I need to pay logical attention to each word, word by word.

Re: Notes on OpenAI's new o1 chain-of-thought models

#397

Post by steveBK123 »

Martinussen wrote: post #388
steveBK123 wrote: post #355

Earlier quoted context omitted.

Indeed. My expectation of a good intern is to produce nothing I will put in production, but show aptitude worth hiring them for. It's a 10 week extended interview with lots of social events, team building, tech talks, presentations, etc. Which is why I've liked the LLM analogy of "unlimited free interns".. I just think some people read that the exact opposite way I do (not very useful).

If I had to respect the basic human rights of my LLM backends, it would probably be less appealing - but "Unlimited free smart-for-being-braindead zombies" might be a little more useful, at least?

Interns, at least on paper, have the optionality of getting better with time in observable obvious ways as they become grad hires, junior engineers, mid engineers etc.

So far, 2 years of publicly accessible LLMs have not improved for intern replacement tasks at the rate a top 50% intern would be expected to.

Re: Notes on OpenAI's new o1 chain-of-thought models

#398

Post by fragmede »

layer8 wrote: post #384
MPSimmons wrote: post #106

Earlier quoted context omitted.

The failure is in how you're using it. I don't mean this as a personal attack, but more to shed light on what's happening. A lot of people use LLMs as a search engine. It makes sense - it's basically a lossy compressed database of everything its ever read, and it generates output that is statistically likely - varying degrees of likeliness depending on the temperature, as well as how many times the particular weights…

> Treat it as a naive but intelligent intern. Provide it data, give it a task, and let it surprise you with its output. Well, I am a naive but intelligent intern (well, senior developer). So in this framing, the LLM can’t do more than I can already do by myself, and thus far it’s very hit or miss if I actually save time, having to provide all the context and requirements, and having to double-check the results. With…

Ooh yeah it's great for bouncing ideas on what to name things off of. You can give it something's function and a backstory and it'll come up with a list of somethings for you to pick and choose from.

Re: Notes on OpenAI's new o1 chain-of-thought models

#399

Post by yreg »

yreg wrote: post #264
eigenket wrote: post #238

Earlier quoted context omitted.

"When the doctor sees the boy he says" Indicates the gender of the father.

A mother can have a male gender. I wonder if this interpretation is a result of attempts to make the model more inclusive than the corpus text, resulting in a guess that's unlikely, but not strictly impossible.

Now I wonder which side is angry about my comment.

Re: Notes on OpenAI's new o1 chain-of-thought models

#400

Post by dartos »

cdelsolar wrote: post #107
elicksaur wrote: post #62

Earlier quoted context omitted.

What games have people made with ChatGPT? Do you have an example of a live, deployed game?

Yes, a gazillion of them. Someone in a scrabble Facebook group made this entirely with ChatGPT: https://aboocher.github.io/scrabble/ingpractice.html

1 I think the problem here is different expectations for what a “game” is.

If you tell a room full of programmers that something can make a game they’re going to expect more than that.

I look at that and I don’t really see a game, I see flashcards.

Still pretty cool chatgpt can put that together.

Also the “try again” button doesn’t work.

Post reply on HN