Learning to Reason with LLMs • Hacker News Board

Re: Learning to Reason with LLMs

#351

Post by worstspotgain » Thu, Sep 12, 2024, 6:48 PM UTC

ARandumGuy wrote: post #137

One thing that makes me skeptical is the lack of specific labels on the first two accuracy graphs. They just say it's a "log scale", without giving even a ballpark on the amount of time it took. Did the 80% accuracy test results take 10 seconds of compute? 10 minutes? 10 hours? 10 days? It's impossible to say with the data they've given us. The coding section indicates "ten hours to solve six challenging algorithmic…

I don't think it's hard to compute the following:

- At the high end, there is a likely nonlinear relationship between answer quality and compute.

- We've gotten used to a flat-price model. With AGI-level models, we might have to pay more for more difficult and more important queries. Such is the inherent complexity involved.

- All this stuff will get better and cheaper over time, within reason.

I'd say let's start by celebrating that machine thinking of this quality is possible at all.

Re: Learning to Reason with LLMs

#352

Post by slashdave » Thu, Sep 12, 2024, 6:48 PM UTC

thelastparadise wrote: post #41

dinobones wrote: post #4

Generating more "think out loud" tokens and hiding them from the user... Idk if I'm "feeling the AGI" if I'm being honest. Also... telling that they choose to benchmark against CodeForces rather than SWE-bench.

Why not? Isn't that basically what humans do? Sit there and think for a while before answering, going down different branches/chains of thought?

Without a world model, not really.

Re: Learning to Reason with LLMs

#353

Post by alok-g » Thu, Sep 12, 2024, 6:49 PM UTC

For the exam problems it gets wrong, has someone cross-checked that the ground truth answers are actually correct!! ;-) Just kidding, but even such a time may come when the exams created by humans start falling short.

Re: Learning to Reason with LLMs

#354

Post by ben_w » Thu, Sep 12, 2024, 6:50 PM UTC

jsheard wrote: post #325

swatcoder wrote: post #251

Earlier quoted context omitted.

> Did the 80% accuracy test results take 10 seconds of compute? 10 minutes? 10 hours? 10 days? It's impossible to say with the data they've given us. The gist of the answer is hiding in plain sight: it took so long, on an exponential cost function, that they couldn't afford to explore any further. The better their max demonstrated accuracy, the more impressive this report is. So why stop where they did? Why omit actu…

See also: them still sitting on Sora seven months after announcing it. They've never given any indication whatsoever of how much compute it uses, so it may be impossible to release in its current state without charging an exorbitant amount of money per generation. We do know from people who have used it that it takes between 10 and 20 minutes to render a shot, but how much hardware is being tied up during that time i…

Could well be.

It's also entirely possible they are simply sincere about their fear it may be used to influence the upcoming US election.

Plenty of people (me included) are sincerely concerned about the way even mere still image generators can drown out the truth with a flood of good-enough-at-first-glance fiction.

Re: Learning to Reason with LLMs

#355

Post by amelius » Thu, Sep 12, 2024, 6:50 PM UTC

valine wrote: post #8

The model performance is driven by chain of thought, but they will not be providing chain of thought responses to the user for various reasons including competitive advantage. After the release of GPT4 it became very common to fine-tune non-OpenAI models on GPT4 output. I’d say OpenAI is rightly concerned that fine-tuning on chain of thought responses from this model would allow for quicker reproduction of their resu…

> I'd say OpenAI is rightly concerned that fine-tuning on chain of thought responses from this model would allow for quicker reproduction of their results.

Why? They're called "Open" AI after all ...

Re: Learning to Reason with LLMs

#356

Post by gloryjulio » Thu, Sep 12, 2024, 6:50 PM UTC

jsheard wrote: post #325

swatcoder wrote: post #251

Earlier quoted context omitted.

> Did the 80% accuracy test results take 10 seconds of compute? 10 minutes? 10 hours? 10 days? It's impossible to say with the data they've given us. The gist of the answer is hiding in plain sight: it took so long, on an exponential cost function, that they couldn't afford to explore any further. The better their max demonstrated accuracy, the more impressive this report is. So why stop where they did? Why omit actu…

See also: them still sitting on Sora seven months after announcing it. They've never given any indication whatsoever of how much compute it uses, so it may be impossible to release in its current state without charging an exorbitant amount of money per generation. We do know from people who have used it that it takes between 10 and 20 minutes to render a shot, but how much hardware is being tied up during that time i…

Also the the sora videos are proven to be modified ads. We still need to see how it perform first

Re: Learning to Reason with LLMs

#357

Post by RockRobotRock » Thu, Sep 12, 2024, 6:51 PM UTC

Shit, this is going to completely kill jailbreaks isn't it?

Re: Learning to Reason with LLMs

#358

Post by guluarte » Thu, Sep 12, 2024, 6:51 PM UTC

the only benchmark that matters in the ELO points on LLMsys, any other one can be easily gamed

Re: Learning to Reason with LLMs

#359

Post by carabiner » Thu, Sep 12, 2024, 6:51 PM UTC

tylervigen wrote: post #293

Here's the o1-preview answer to the strawberry question: -- There are *three* letter "R"s in the word "strawberry." Let's break down the word to count the occurrences: - *S* - *T* - *R* - *A* - *W* - *B* - *E* - *R* - *R* - *Y* The letter "R" appears in positions 3, 8, and 9.

can you ask it:

9.11 and 9.9, which number is larger

Re: Learning to Reason with LLMs

#360

Post by zamadatix » Thu, Sep 12, 2024, 6:52 PM UTC

wmf wrote: post #184

ARandumGuy wrote: post #137

One thing that makes me skeptical is the lack of specific labels on the first two accuracy graphs. They just say it's a "log scale", without giving even a ballpark on the amount of time it took. Did the 80% accuracy test results take 10 seconds of compute? 10 minutes? 10 hours? 10 days? It's impossible to say with the data they've given us. The coding section indicates "ten hours to solve six challenging algorithmic…

People have been celebrating the fact that tokens got 100x cheaper and now here's a new system that will use 100x more tokens.

The new thing that can do more at the "ceiling" price doesn't remove your ability to still use the 100x cheaper tokens for the things that were doable on that version.