Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning to Reason with LLMs

openai.com

Post reply on HN
Print view View original

251–260 of 1001 posts

Re: Learning to Reason with LLMs

#251

Post by swatcoder »

ARandumGuy wrote: post #137

One thing that makes me skeptical is the lack of specific labels on the first two accuracy graphs. They just say it's a "log scale", without giving even a ballpark on the amount of time it took. Did the 80% accuracy test results take 10 seconds of compute? 10 minutes? 10 hours? 10 days? It's impossible to say with the data they've given us. The coding section indicates "ten hours to solve six challenging algorithmic…

> Did the 80% accuracy test results take 10 seconds of compute? 10 minutes? 10 hours? 10 days? It's impossible to say with the data they've given us.

The gist of the answer is hiding in plain sight: it took so long, on an exponential cost function, that they couldn't afford to explore any further.

The better their max demonstrated accuracy, the more impressive this report is. So why stop where they did? Why omit actual clock times or some cost proxy for it from the report? Obviously, it's because continuing was impractical and because those times/costs were already so large that they'd unfavorably affect how people respond to this report

Re: Learning to Reason with LLMs

#252

Post by [deleted] »

seydor wrote: post #241
wmf wrote: post #184

Earlier quoted context omitted.

People have been celebrating the fact that tokens got 100x cheaper and now here's a new system that will use 100x more tokens.

If it 's reasoning correctly, it shouldnt need a lot of tokens because you don't need to correct it. You only need to ask it to solve nuclear fusion once.

[deleted]

Re: Learning to Reason with LLMs

#253

Post by mewpmewp2 »

extr wrote: post #151

Interesting that the coding win-rate vs GPT-4o was only 10% higher. Very cool but clearly this model isn't as much of a slam dunk as the static benchmarks portray. However, it does open up an interesting avenue for the future. Could you prompt-cache just the chain-of-thought reasoning bits?

It's hard to evaluate those win-rates, because if it's slower, people may have been giving easier problems, which both can solve and picked the faster one.

Re: Learning to Reason with LLMs

#254

Post by tcdent »

valine wrote: post #8

The model performance is driven by chain of thought, but they will not be providing chain of thought responses to the user for various reasons including competitive advantage. After the release of GPT4 it became very common to fine-tune non-OpenAI models on GPT4 output. I’d say OpenAI is rightly concerned that fine-tuning on chain of thought responses from this model would allow for quicker reproduction of their resu…

CoT is now their primary method for alignment. Exposing that information would negate that benefit.

I don't agree with this, but it definitely carries higher weight in their decision making than leaking relevant training info to other models.

Re: Learning to Reason with LLMs

#255

Post by ansc »

apsec112 wrote: post #152
cal85 wrote: post #54

Sounds great, but so does their "new flagship model that can reason across audio, vision, and text in real time" announced in May. [0] [0] https://openai.com/index/hello-gpt-4o/

This one [o1/Strawberry] is available. I have it, though it's limited to 30 messages/week in ChatGPT Plus.

30 messages per week? Wow. You better not miss!

Re: Learning to Reason with LLMs

#256

Post by msp26 »

seydor wrote: post #241
wmf wrote: post #184

Earlier quoted context omitted.

People have been celebrating the fact that tokens got 100x cheaper and now here's a new system that will use 100x more tokens.

If it 's reasoning correctly, it shouldnt need a lot of tokens because you don't need to correct it. You only need to ask it to solve nuclear fusion once.

Have you seen how long the CoT was for the example. It's incredibly verbose.

Re: Learning to Reason with LLMs

#257

Post by changoplatanero »

evrydayhustling wrote: post #243
changoplatanero wrote: post #227

Earlier quoted context omitted.

Hmm, are you sure it was using the o1 model and not gpt4o? I've been using the o1 model and it does consistently well at solving rotation ciphers.

o1-preview . Were you using common plaintexts by chance (e.g. proverbs) or ROT13 specifically? Mine use all the right steps but just can't string them together.

It just solved this one for me "Can you solve this cipher? droboscxylecsxoccvsuocryglecsxocc"

Re: Learning to Reason with LLMs

#259

Post by seydor »

afro88 wrote: post #186
Hansenq wrote: post #87

Reading through the Chain of Thought for the provided Cipher example (go to the example, click "Show Chain of Thought") is kind of crazy...it literally spells out every thinking step that someone would go through mentally in their head to figure out the cipher (even useless ones like "Hmm"!). It really seems like slowing down and writing down the logic it's using and reasoning over that makes it better at logic, simi…

Seeing the "hmmm", "perfect!" etc. one can easily imagine the kind of training data that humans created for this. Being told to literally speak their mind as they work out complex problems.

looks a bit like 'code', using keywords 'Hmm', 'Alternatively', 'Perfect'

Re: Learning to Reason with LLMs

#260

Post by RobertDeNiro »

mewpmewp2 wrote: post #199
not_pleased wrote: post #190

The progress in AI is incredibly depressing, at this point I don't think there's much to look forward to in life. It's sad that due to unearned hubris and a complete lack of second-order thinking we are automating ourselves out of existence. EDIT: I understand you guys might not agree with my comments. But don't you thinking that flagging them is going a bit too far?

It seems opposite to me. Imagine all the amazing technological advancements, etc. If there wasn't something like that what would you be looking forward to? Everything would be what it has already been for years. If this evolves it helps us open so many secrets of the universe.

These advancements are there to benefit the top 1%, not the working class.
Post reply on HN