Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning to Reason with LLMs

openai.com

Post reply on HN
Print view View original

481–490 of 1001 posts

Re: Learning to Reason with LLMs

#481

Post by oytis »

hidelooktropic wrote: post #423
carabiner wrote: post #359

Earlier quoted context omitted.

can you ask it: 9.11 and 9.9, which number is larger

> Evaluating Number formats > I'm considering that the question might be about comparing two numbers, 9.11 and 9.9, in different contexts such as different locale formats or representing dates. 9.11 is larger than 9.9 Well, it's kind of right because we left it ambiguous what kinds of numbers these are. September 11 is a greater date than September 9. However, I would want the model to have the street smarts to assum…

Was it trained on Excel code?

Re: Learning to Reason with LLMs

#482

Post by bluecoconut »

bluecoconut wrote: post #417
bluecoconut wrote: post #339

Earlier quoted context omitted.

Super hand-waving rough estimate: Going off of five points of reference / examples that sorta all point in the same direction. 1. looks like they scale up by about ~100-200 on the x axis when showing that test time result. 2. Based on the o1-mini post [1], there's an "inference cost" where you can see GPT-4o and GPT-4o mini as dots in the bottom corner, haha (you can extract X values, ive done so below) 3. There's a…

Some other follow up reflections 1. I wish that Y-axes would switch to be logit instead of linear, to help see power-law scaling on these 0->1 measures. In this case, 20% -> 80% it doesn't really matter, but for other papers (eg. [2] below) it would help see this powerlaw behavior much better. 2. The power law behavior of inference compute seems to be showing up now in multiple ways. Both in ensembles [1,2], as well…

oh, they do talk about it

  On the 2024 AIME exams, GPT-4o only solved on average 12% (1.8/15) of problems. o1 averaged 74% (11.1/15) with a single sample per problem, 83% (12.5/15) with consensus among 64 samples, and 93% (13.9/15) when re-ranking 1000 samples with a learned scoring function. A score of 13.9 places it among the top 500 students nationally and above the cutoff for the USA Mathematical Olympiad.
showing that as they increase the k of ensemble, they can continue to get it higher. All the way up to 93% when using 1000 samples.

Re: Learning to Reason with LLMs

#483

Post by Workaccount2 »

hakanderyal wrote: post #418
cs391231 wrote: post #408

Student here. Can someone give me one reason why I should continue in software engineering that isn't denial and hopium?

Software engineering contains a lot more than just writing code. If we somehow get AGI, it'll change everything, not just SWE. If not, my belief is that there will be a lot more demand for good SWEs to harness the power of LLMs, not less. Use them to get better at it faster.

I don't think anyone is worried about SWE work going away, I think the concern is if SWE's will still be able to command cushy salaries and working conditions.

Re: Learning to Reason with LLMs

#484

Post by _yb2s »

slashdave wrote: post #352
thelastparadise wrote: post #41

Earlier quoted context omitted.

Why not? Isn't that basically what humans do? Sit there and think for a while before answering, going down different branches/chains of thought?

Without a world model, not really.

The whole thing is a world model- accurately predicting text that describes things happening in a world, can only be done by modeling the world.

Re: Learning to Reason with LLMs

#485

Post by LarsDu88 »

I wonder if this architecture is just asking a chain of thought prompt, or whether they built a diffusion model.

The old problem with image generation was that single pass techniques like GANs and VAEs had to do everything in one go. Diffusion models wound up being better by doing things iteratively.

Perhaps this is a diffusion model for text (top ICML paper this year was related to this).

Re: Learning to Reason with LLMs

#486

Post by stickfigure »

cs391231 wrote: post #408

Student here. Can someone give me one reason why I should continue in software engineering that isn't denial and hopium?

The amount of knowledge the OP needed to be even to formulate the right question to the AI requires a lifetime of deep immersion in technology. You'd think that maybe you can ask the AI how to phrase the question to the AI but at some point you run up against your ability to contextualize the problem - it can't read your mind.

Will the AI become as smart as you or I? Recognize that these things have tiny context windows. You get the context window of "as long as you can remember".

I don't see this kind of AI replacing programmers (though it probably will replace low-skill offshore contract shops). It may have a large magnifying effect on skill. Fortunately there seem to be endless problems to solve with software - it's not like bridges or buildings; you only need (or can afford) so many. Architects should probably be more worried.

Re: Learning to Reason with LLMs

#487

Post by startupsfail »

cs391231 wrote: post #408

Student here. Can someone give me one reason why I should continue in software engineering that isn't denial and hopium?

The timeline to offload SWE tasks to AI is likely 5+ years. So there are still some years left before the exchange of a “brain on a stick” for “property and material goods” would become more competitive and demanding because of direct AI competition.

Re: Learning to Reason with LLMs

#489

Post by peppertree »

octernion wrote: post #410
carabiner wrote: post #359

Earlier quoted context omitted.

can you ask it: 9.11 and 9.9, which number is larger

it claims 9.11 is larger, hah. https://postimg.cc/KKKLZNjW

That's odd. Tried several times with the same prompt and always got the correct answer with step by step explanation.
Post reply on HN