Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning to Reason with LLMs

openai.com

Post reply on HN
Print view View original

871–880 of 1001 posts

Re: Learning to Reason with LLMs

#871

Post by matwood »

Workaccount2 wrote: post #483
hakanderyal wrote: post #418

Earlier quoted context omitted.

Software engineering contains a lot more than just writing code. If we somehow get AGI, it'll change everything, not just SWE. If not, my belief is that there will be a lot more demand for good SWEs to harness the power of LLMs, not less. Use them to get better at it faster.

I don't think anyone is worried about SWE work going away, I think the concern is if SWE's will still be able to command cushy salaries and working conditions.

Salaries will only change if tech loses it's leverage on the economy. Think of it this way, if Google can operate Google with only 10% of its current staff, then there will be other Googles popping up. The downward pressure on salaries will start with the downward pressure on tech overall. I'm not sure I see this happening anytime soon because humanity is so good at using every resource available.

Re: Learning to Reason with LLMs

#872

Post by mr_toad »

almostuseful wrote: post #596
cs391231 wrote: post #408

Student here. Can someone give me one reason why I should continue in software engineering that isn't denial and hopium?

If at some point a competent senior software engineer can be automated away, I think we are so close to a possible 'AI singularity' in as much as that concept makes sense, that nothing really matters anyway. I don't know what will be automated first of the competent senior software engineer and say, a carpenter, but once the programmer has been automated away, the carpenter (and everything else) will follow shortly.…

Carpenters and other manual jobs might outlast software engineers. It seems that AI is advancing a lot faster than robotics.

Re: Learning to Reason with LLMs

#873

Post by energy123 »

wmf wrote: post #184
ARandumGuy wrote: post #137

One thing that makes me skeptical is the lack of specific labels on the first two accuracy graphs. They just say it's a "log scale", without giving even a ballpark on the amount of time it took. Did the 80% accuracy test results take 10 seconds of compute? 10 minutes? 10 hours? 10 days? It's impossible to say with the data they've given us. The coding section indicates "ten hours to solve six challenging algorithmic…

People have been celebrating the fact that tokens got 100x cheaper and now here's a new system that will use 100x more tokens.

It does dispel this idea that we are going to be flooded with too many GPUs.

Re: Learning to Reason with LLMs

#874

Post by takoid »

joshhug wrote: post #854

I just tried o1, and it did pretty well with understanding this minor issue with subtitles on a Dutch TV show we were watching. I asked it "I was watching a show and in the subtitles an umlaut u was rendered as 1/4, i.e. a single character that said 1/4. Why would this happen?" and it gave a pretty thorough explanation of exactly which encoding issue was to blame. https://chatgpt.com/share/66e37145-72bc-800a-be7b-f7c…

4o’s answer seems sufficient, though it provides less detail than o1.

https://chatgpt.com/share/66e373d7-7814-8009-86c3-1ce549ca2e...

Re: Learning to Reason with LLMs

#875

Post by karmasimida »

joshhug wrote: post #854

I just tried o1, and it did pretty well with understanding this minor issue with subtitles on a Dutch TV show we were watching. I asked it "I was watching a show and in the subtitles an umlaut u was rendered as 1/4, i.e. a single character that said 1/4. Why would this happen?" and it gave a pretty thorough explanation of exactly which encoding issue was to blame. https://chatgpt.com/share/66e37145-72bc-800a-be7b-f7c…

Damn, the model really goes to length to those trivial but hard problems. Impressive

Re: Learning to Reason with LLMs

#876

Post by m348e912 »

OutOfHere wrote: post #849
m348e912 wrote: post #642

I have a straight forward task that no model has been able to successfully complete. The request is pretty basic. If anyone can get it to work, I'd like to know how and what model you're using. I tried it with gpt4o1 and after ~10 iterations of showing it the failed output, it still failed to come up with a one-line command to properly display results. Here it what I asked: Using a mac osx terminal and standard avail…

Have you tried `ss -ar`? You may have to install `ss`. It is standard on Linux.

No I was trying to see it could use tools/binaries that come with MacOS.

Re: Learning to Reason with LLMs

#877

Post by RobertDeNiro »

baq wrote: post #662
activatedgeek wrote: post #610

Earlier quoted context omitted.

Reasoning tokens are indeed billed as output tokens. > While reasoning tokens are not visible via the API, they still occupy space in the model's context window and are billed as output tokens. From here: https://platform.openai.com/docs/guides/reasoning

This is concerning - how do you know you aren’t being fleeced out of your money here…? You’ll get your results, but did you really use that much?

Also seems very impractical to embed this into a deployed product. How can you possibly hope to control and estimate costs? I guess this is strictly meant for R&D purposes.

Re: Learning to Reason with LLMs

#878

Post by energy123 »

silveryfu wrote: post #863

After playing with it on ChatGPT this morning, it seems a reasonable strategy of using the o1 model is to: - If your request requires reasoning, switch to o1 model. - If not, switch to 4o model. This applies to both across chat sessions and within the same session (yes, we can switch between models within the same session and it looks like down the road OpenAI is gonna support automatic model switching). Based on my…

This was mentioned in OpenAI's report. People rated o1 as the same or worse than GPT-4o if the prompt didn't require reasoning, like on personal writing tasks.

Re: Learning to Reason with LLMs

#879

Post by scotty79 »

andrewla wrote: post #62

This is something that people have toyed with to improve the quality of LLM responses. Often instructing the LLM to "think about" a problem before giving the answer will greatly improve the quality of response. For example, if you ask it how many letters are in the correctly spelled version of a misspelled word, it will first give the correct spelling, and then the number (which is often correct). But if you instruct…

> I like the idea too that they turbocharged it by taking the limits off during the "thinking" state

They also specifically trained the model to do that thinking out loud.

Re: Learning to Reason with LLMs

#880

Post by gwern »

dinobones wrote: post #4

Generating more "think out loud" tokens and hiding them from the user... Idk if I'm "feeling the AGI" if I'm being honest. Also... telling that they choose to benchmark against CodeForces rather than SWE-bench.

> Also... telling that they choose to benchmark against CodeForces rather than SWE-bench.

They also worked with Devin to benchmark it on Devin's internal benchmarks, where it's twice as good as GPT-4o: https://x.com/cognition_labs/status/1834292718174077014 https://www.cognition.ai/blog/evaluating-coding-agents

Post reply on HN