Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning to Reason with LLMs

openai.com

Post reply on HN
Print view View original

171–180 of 1001 posts

Re: Learning to Reason with LLMs

#171

Post by Jasper_ »

Hansenq wrote: post #87

Reading through the Chain of Thought for the provided Cipher example (go to the example, click "Show Chain of Thought") is kind of crazy...it literally spells out every thinking step that someone would go through mentally in their head to figure out the cipher (even useless ones like "Hmm"!). It really seems like slowing down and writing down the logic it's using and reasoning over that makes it better at logic, simi…

> Average:18/2=9

> 9 corresponds to 'i'(9='i')

> But 'i' is 9, so that seems off by 1.

Still seems bad at counting, as ever.

Re: Learning to Reason with LLMs

#172

Post by jslakro »

TheAceOfHearts wrote: post #99
zozbot234 wrote: post #49

Earlier quoted context omitted.

> Hopefully the open source community will figure out how to effectively match and replicate what OpenAI is doing. No need for that, there is a Reflection 70B model that does the exact same thing - with chains of thought being separated from the "final answer" via custom 'tag' tokens.

Wasn't this the model that was proven to have been faking their benchmarks recently? Or am I thinking of a different model?

It's the same, for sure the proximity of that little scandal to this announcement is no coincidence.

Re: Learning to Reason with LLMs

#173

Post by bluecoconut »

arresin wrote: post #119

> Unless otherwise specified, we evaluated o1 on the maximal test-time compute setting. Maximal test time is the maximum amount of time spent doing the “Chain of Thought” “reasoning”. So that’s what these results are based on. The caveat is that in the graphs they show that for each increase in test-time performance, the (wall) time / compute goes up exponentially . So there is a potentially interesting play here. Th…

This power law behavior of test-time improvement seems to be pretty ubiquitous now. In more agents is all you need [1], they start to see this as a function of ensemble size. It also shows up in: Large Language Monkeys: Scaling Inference Compute with Repeated Sampling [2]

I sorta wish everyone would plot their y-axis with logit y-axis, rather than 0->100 accuracy (including the openai post), to help show the power-law behavior. This is especially important when talking about incremental gains in the ~90->95, 95->99%. When the values (like the open ai post) are between 20->80, logit and linear look pretty similar, so you can "see" the inference power-law

[1] https://arxiv.org/abs/2402.05120 [2] https://arxiv.org/abs/2407.21787

Re: Learning to Reason with LLMs

#174

Post by rvnx »

apsec112 wrote: post #108
benterix wrote: post #103

Earlier quoted context omitted.

Read "immediate" in "immediate use" in the same way as "open" in "OpenAI".

You can use it, I just tried a few minutes ago. It's apparently limited to 30 messages/week, though.

The option isn't there for us (though the blogpost says otherwise), even after CTRL-SHIFT-R, hence the parent comment.

Re: Learning to Reason with LLMs

#175

Post by adverbly »

paxys wrote: post #110
flockonus wrote: post #85

Are we ready yet to admit Turing test has been passed?

The Turing Test (which involves fooling a human into thinking they are talking to another human rather than a computer) has been routinely passed by very rudimentary "AI" since as early as 1991. It has no relevance today.

This is only true for some situations. In some test conditions it has not been passed. I can't remember the exact name, but there used to be a competition where PhD level participants blindly chat for several minutes with each other and are incentivized to discover who is a bot and who is a human. I can't remember if they still run it, but that bar has never been passed from what I recall.

Re: Learning to Reason with LLMs

#178

Post by riazrizvi »

I’m not surprised there’s no comparison to GPT-4. Was 4o a rewrite on lower specced hardware and a more quantized model, where the goal was to reduce costs while trying to maintain functionality? Do we know if that is so? That’s my guess. If so is O1 an upgrade in reasoning complexity that also runs on cheaper hardware?

Re: Learning to Reason with LLMs

#179

Post by baq »

andrewla wrote: post #42
tomtom1337 wrote: post #15

Earlier quoted context omitted.

Can you explain what you mean by this?

This is a transcription of a literal quote from the article: > Therefore, after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users

At least they're open about not being open. Very meta OpenAI.

Re: Learning to Reason with LLMs

#180

Post by Filligree »

TheAceOfHearts wrote: post #99
zozbot234 wrote: post #49

Earlier quoted context omitted.

> Hopefully the open source community will figure out how to effectively match and replicate what OpenAI is doing. No need for that, there is a Reflection 70B model that does the exact same thing - with chains of thought being separated from the "final answer" via custom 'tag' tokens.

Wasn't this the model that was proven to have been faking their benchmarks recently? Or am I thinking of a different model?

That’s the one.
Post reply on HN