Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on OpenAI's new o1 chain-of-thought models

simonwillison.net

Post reply on HN
Print view View original

451–460 of 659 posts

Re: Notes on OpenAI's new o1 chain-of-thought models

#451

Post by HarHarVeryFunny »

I think Rich Sutton's bitter lesson will prove to apply here, and what we really need to advance machine learning capabilities are more general and powerful models capable of learning for themselves - better able to extract and use knowledge from the firehose of data available from the real world (ultimately via some form of closed-loop deployment where they can act and incrementally learn from their own actions).

What OpenAI have delivered here is basically a hack - a neuro-symbolic agent that has a bunch of hard-coded "reasoning" biases built in (via RL). It's a band-aid approach to try to provide some of what's missing from the underlying model which was never designed for what it's now being asked to do.

Re: Notes on OpenAI's new o1 chain-of-thought models

#452

Post by ai4ever »

henry2023 wrote: post #103

>> which is larger 3.9, 3.10 or 3.11 To determine which number is larger among 3.9, 3.10, and 3.11, let's compare Standard Decimal Interpretation: 3.9 is simply 3.9. 3.10 is equivalent to 3.1 because trailing zeros after the decimal point do not add value in decimal numbers. 3.11 remains 3.11. Comparison: 3.9 vs. 3.1 vs. 3.11 3.11 is the largest number. Version Number Interpretation: In version numbering, each segmen…

lol,

they gamed AIME by over-training the hell out of it for marketing purposes and called it done.

meanwhile, back-to-basics is broken.

Re: Notes on OpenAI's new o1 chain-of-thought models

#453

Post by joelburget »

HarHarVeryFunny wrote: post #451

I think Rich Sutton's bitter lesson will prove to apply here, and what we really need to advance machine learning capabilities are more general and powerful models capable of learning for themselves - better able to extract and use knowledge from the firehose of data available from the real world (ultimately via some form of closed-loop deployment where they can act and incrementally learn from their own actions). Wh…

o1 is an application of the Bitter Less. To quote Sutton: "The two methods that seem to scale arbitrarily in this way are search and learning." (emphasis mine -- in the original Sutton also emphasized learning).

OpenAI and others have previously pushed the learning side, while neglecting search. Now that gains from adding compute at training time have started to level off, they're adding compute at inference time.

Re: Notes on OpenAI's new o1 chain-of-thought models

#454

Post by ai4ever »

ldjkfkdsjnv wrote: post #5

The theory is that this solves the data shortage problem, they can generate a ton of chain of reasoning data from what we already have. True iterative improvement, like out of a science fiction novel These models are going to get embedded deeply into IDE's, like cursor has, and essentially end software development as we know it. A properly written requirements spec, and an engineer, can do the work of 5. Software eng…

Such statements are made by management folks who dont code, and somehow think coding can be hand-waved away.

Sure, this tool will improve the productivity of sw engineers, but so did the compiler which came 50 years back.

Re: Notes on OpenAI's new o1 chain-of-thought models

#455

Post by EGreg »

bambax wrote: post #281

Near the end, the quote from OpenAI researcher Jason Wei seems damning to me: > Results on AIME and GPQA are really strong, but that doesn’t necessarily translate to something that a user can feel. Even as someone working in science, it’s not easy to find the slice of prompts where GPT-4o fails, o1 does well, and I can grade the answer. But when you do find such prompts, o1 feels totally magical. We all need to find…

Dont you know by now

Speaking with AI maxis it’s easy:

The AI is always right

You are always wrong

If AI might enable something dangerous, it was already possible by hand, scale is irrelevant

But also AI enables many amazing things not previously possible, at scale

If you don’t get the answers you want, you’re prompting it wrong. You need to work harder to show how much better the AI is. But definitely, it cannot make things worse at scale in any way. And anyone who wants regulations to even require attribution and labeling, is a dangerous luddite depriving humanity of innovations.

Re: Notes on OpenAI's new o1 chain-of-thought models

#456

Post by freediver »

oersted wrote: post #265
freediver wrote: post #141

Not seeing major advance in quality with o1, but seeing major negative impact on cost and latency. Kagi LLM benchmarking project: https://help.kagi.com/kagi/ai/llm-benchmark.html

Kagi is most likely evaluating it mainly on deriving an answer for the user from search result snippets. Indeed, GPT-4o is plenty good at this already, and o1 would only perform better on particular types of hard requests, while being so much slower. If you look at Appendix A in the o1 post [1], this becomes quite clear. There's a huge jump in performance in "puzzle" tasks like competitive maths or programming. But t…

The test has many reasoning, code and instruction following questions which I expected o1 to be excelling at. I do not have an interpretation for such poor results on our test, was just sharing them as a data point for people to make their own mind. My best guess at this point is that o1 is optimized for a very specific and narrow use case, similar to what you suggest.

Re: Notes on OpenAI's new o1 chain-of-thought models

#457

Post by nbzso »

Working in tech for over 30 years. This is the first time when I don't see proposed technology as a valuable tool. Especially LLM's. Vastly overhyped, driven by pure greed and speculative narratives, limited implementation and high energy cost. Non-transparent. Errors marketed as a hallucination.

Re: Notes on OpenAI's new o1 chain-of-thought models

#458

Post by tptacek »

jeffhuys wrote: post #169
INGSOCIALITE wrote: post #82

Earlier quoted context omitted.

it's all placeholders - that's my experience with gpt trying to write slop code

Then ask it to expand. Be specific.

I wasn't about to paste 1000 lines of Python into the thread; I just picked an interesting snippet.

Re: Notes on OpenAI's new o1 chain-of-thought models

#459

Post by simonw »

nbzso wrote: post #457

Working in tech for over 30 years. This is the first time when I don't see proposed technology as a valuable tool. Especially LLM's. Vastly overhyped, driven by pure greed and speculative narratives, limited implementation and high energy cost. Non-transparent. Errors marketed as a hallucination.

How much time have you spent figuring out how to use them?

Ethan Mollick estimates it takes ten hours of exposure to “frontier models” (aka OpenAI GPT-4, Claude 3.5 Sonnet, Google Gemini 1.5 Pro) before they really start to click in terms of what they’re useful for.

Re: Notes on OpenAI's new o1 chain-of-thought models

#460

Post by TeMPOraL »

swader999 wrote: post #426
idiotsecant wrote: post #391

Earlier quoted context omitted.

I think LLMs are definitely on the path to AGI in the same way that the ball bearing was on the path to the internal combustion engine. I think its quite likely that LLMs will perform important functions within the system of an eventual AGI.

This may be accurate. I wonder if there's enough energy in the world for this endeavour.

Of course!

1. We've barely scratched the surface of this solution space; the focus only recently started shifting from improving model capabilities to improving training costs. People are looking at more efficient architectures, and lots of money is starting to flow in that direction, so it's a safe bet things will get significantly more efficient.

2. Training is expensive, inference is cheap, copying is free. While inference costs add up with use, they're still less than costs of humans doing the equivalent work, so out of all things AI will impact, I wouldn't worry about energy use specifically.

Post reply on HN