Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

futurism.com

Post reply on HN
Print view View original

41–50 of 174 posts

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#41

Post by caconym_ »

chasing0entropy wrote: post #2

The models were restricted from accessing the internet and forced to develop their own solutions internally. I think researchers will find that human coders are unable to solve most coding problems without access to the internet.

A decent human programmer with experience in a particular domain may rely on internet access to look up API documentation and other generic references, but if you read the paper, you'll see that the AI systems tested suffered from more basic deficiencies in approach and reasoning ('3.6. Discussion', starting on page 7).

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#42

Post by rurp »

pzo wrote: post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

It depends a lot on the type of problem. If we're talking about fixing a bug or adding a new feature to a large existing code base, which probably describes a huge portion or professional software engineering work, I would say most engineers could do most of those tasks without the internet. Especially if the goal is to simply pass a benchmark test of getting it working without future considerations.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#43

Post by inykt »

pzo wrote: post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

These models are held at higher standards than humans. They should be above to solve any coding problem with just the documentation.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#44

Post by petesergeant »

jasonthorsness wrote: post #19

Half of the work is specification and iteration. I think there’s a focus on full SWE replacement because it’s sensational, but we’ll more end up with SWE able to focus on the less patterned or ambiguous work and made way more productive with the LLM handling subtasks more efficiently. I don’t see how full SWE replacement can happen unless non-SWE people using LLMs become technical enough to get what they need out of…

> in which case they probably have just become SWE anyway

or learn to use something like Bubble

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#45

Post by achierius »

pzo wrote: post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

Isn't this how interviews tend to work? So I think a good number of devs would, yes.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#46

Post by rpmisms »

ITB wrote: post #28
mrayycombi wrote: post #11

Earlier quoted context omitted.

Where are the low level CEOs vs high level CEOs? I'll bet AI could do their jobs right now. Can SOMEONE please write AI software to replace these people?

Are you bothered by the fact that software engineers might be easier to automate?

Considering that there are chickens who outperform stockbrokers, no.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#47

Post by kristopolous »

pzo wrote: post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

I used to be able to. The web's made me lazy. I was better before it and I'm better when I don't use it.

Really, the stuff you think helps you is often just holding you back.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#48

Post by anonymoushn »

_def wrote: post #4

> even though CEO Sam Altman insists they will be able to beat "low-level" software engineers by the end of this year. "low/high level" starts to lose its meaning to me because it gets used in opposite ways

Even better, if you click through to the linked source he doesn't say "low-level" at all, or make any claim that is at all like the claim he is cited as making!

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#49

Post by thrw011 »

pzo wrote: post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

> How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)?

Many, there was a time when SO did not exist and people were able to solve non trivial problems. There was a time coding problems on exams had to be solved on paper and if they were not compiling you would not pass.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#50

Post by ipython »

pzo wrote: post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

What would searching the Internet provide the models that they don’t already have? Most likely data sources such as stack overflow, documentation on the language it’s targeting, and a variety of relevant forum posts are already part of its training set.

Unless someone else came along and said “here’s how to solve x problem step by step”, I don’t see how additional information past its cutoff point would help. (Perhaps the AI could post on a forum and wait for an answer?)

Yes, iterative programming could help via access to tools- I can see that helping.

Post reply on HN