Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning to Reason with LLMs

openai.com

Post reply on HN
Print view View original

271–280 of 1001 posts

Re: Learning to Reason with LLMs

#271

Post by packetlost »

ARandumGuy wrote: post #245
packetlost wrote: post #191

Earlier quoted context omitted.

When one axis is on log scale and the other is linear with the plot points appearing linear-ish, doesn't it mean there's a roughly exponential relationship between the two axis?

It'd be more accurate to call it a logarithmic relationship, since compute time is our input variable. Which itself is a bit concerning, as that implies that modest gains in accuracy require exponentially more compute time. In either case, that still doesn't excuse not labeling your axis. Taking 10 seconds vs 10 days to get 80% accuracy implies radically different things on how developed this technology is, and how v…

> Which itself is a bit concerning, as that implies that modest gains in accuracy require exponentially more compute time

This is more of what I was getting at. I agree they should label the axis regardless, but I think the scaling relationship is interesting (or rather, concerning) on its own.

Re: Learning to Reason with LLMs

#272

Post by adverbly »

> Therefore, s(x)=p∗(x)−x2n+2 We can now write, s(x)=p∗(x)−x2n+2

Completely repeated itself... weird... it also says "...more lines cut off..." How many lines I wonder? Would people get charged for these cut off lines? Would have been nice to see how much answer had cost...

Re: Learning to Reason with LLMs

#273

Post by thomasahle »

seydor wrote: post #259
afro88 wrote: post #186

Earlier quoted context omitted.

Seeing the "hmmm", "perfect!" etc. one can easily imagine the kind of training data that humans created for this. Being told to literally speak their mind as they work out complex problems.

looks a bit like 'code', using keywords 'Hmm', 'Alternatively', 'Perfect'

Right, these are not mere "filler words", but initialize specific reasoning paths.

Re: Learning to Reason with LLMs

#274

Post by [deleted] »

apsec112 wrote: post #152
cal85 wrote: post #54

Sounds great, but so does their "new flagship model that can reason across audio, vision, and text in real time" announced in May. [0] [0] https://openai.com/index/hello-gpt-4o/

This one [o1/Strawberry] is available. I have it, though it's limited to 30 messages/week in ChatGPT Plus.

[deleted]

Re: Learning to Reason with LLMs

#276

Post by bjornsing »

ARandumGuy wrote: post #137

One thing that makes me skeptical is the lack of specific labels on the first two accuracy graphs. They just say it's a "log scale", without giving even a ballpark on the amount of time it took. Did the 80% accuracy test results take 10 seconds of compute? 10 minutes? 10 hours? 10 days? It's impossible to say with the data they've given us. The coding section indicates "ten hours to solve six challenging algorithmic…

So now it’s a question of how fast the AGI will run? :)

Re: Learning to Reason with LLMs

#277

Post by seydor »

adverbly wrote: post #237

> However, o1-preview is not preferred on some natural language tasks, suggesting that it is not well-suited for all use cases. Fascinating... Personal writing was not preferred vs gpt4, but for math calculations it was... Maybe we're at the point where its getting too smart? There is a depressing related thought here about how we're too stupid to vote for actually smart politicians ;)

> for actually smart politicians

We can vote an AI

Re: Learning to Reason with LLMs

#278

Post by TillE »

flockonus wrote: post #85

Are we ready yet to admit Turing test has been passed?

Extremely basic agency would be required to pass the Turing test as intended.

Like, the ability to ask a new unrelated question without being prompted. Of course you can fake this, but then you're not testing the LLM as an AI, you're testing a dumb system you rigged up to create the appearance of an AI.

Re: Learning to Reason with LLMs

#279

Post by idiliv »

In the demo, O1 implements an incorrect version of the "squirrel finder" game?

The instructions state that the squirrel icon should spawn after three seconds, yet it spawns immediately in the first game (also noted by the guy doing the demo).

Edit: I'm referring to the demo video here: https://openai.com/index/introducing-openai-o1-preview/

Re: Learning to Reason with LLMs

#280

Post by charlescurt123 »

seydor wrote: post #241
wmf wrote: post #184

Earlier quoted context omitted.

People have been celebrating the fact that tokens got 100x cheaper and now here's a new system that will use 100x more tokens.

If it 's reasoning correctly, it shouldnt need a lot of tokens because you don't need to correct it. You only need to ask it to solve nuclear fusion once.

with these methods the issue is the log scale of compute. Let's say you ask it to solve fusion. It may be able to solve it but the issue is it's unverifiable WHICH was correct.

So it may generate 10 Billion answers to fusion and only 1-10 are correct.

There would be no way to know which one is correct without first knowing the answer to the question.

This is my main issue with these methods. They assume the future via RL then when it gets it right they mark that.

We should really be looking at methods of percentage it was wrong rather then it was right a single time.

Post reply on HN