Live data from Hacker News

OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

futurism.com

11–20 of 174 posts

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#11
post #4

> even though CEO Sam Altman insists they will be able to beat "low-level" software engineers by the end of this year. "low/high level" starts to lose its meaning to me because it gets used in opposite ways

Where are the low level CEOs vs high level CEOs?

I'll bet AI could do their jobs right now.

Can SOMEONE please write AI software to replace these people?

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#14

The models were restricted from accessing the internet and forced to develop their own solutions internally. I think researchers will find that human coders are unable to solve most coding problems without access to the internet.

Damn. How did they write code before 1992?

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#16
post #4

> even though CEO Sam Altman insists they will be able to beat "low-level" software engineers by the end of this year. "low/high level" starts to lose its meaning to me because it gets used in opposite ways

Yeah low-level language conflated with low-level coders , means the opposite in some sense

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#17

The models were restricted from accessing the internet and forced to develop their own solutions internally. I think researchers will find that human coders are unable to solve most coding problems without access to the internet.

What kinds of problems are you talking about? There are problems that require you to learn new libraries and services constantly, accessing documentation, and there are actual software problems when you have to reflect on your own large code base. I work on both kinds of problems, and in the first case, the models are actually well versed in say, all of CloudFormation syntax, that I would have to look up. On the opposite end, I have written many features on trips, unable to be distracted by the internet, just me and the code, and being able to read library source code.

The fact is, programming requires abstract modeling that language models aren’t demonstrating the capability of fully replicating. At least, not that we can see, yet.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#19
Half of the work is specification and iteration. I think there’s a focus on full SWE replacement because it’s sensational, but we’ll more end up with SWE able to focus on the less patterned or ambiguous work and made way more productive with the LLM handling subtasks more efficiently. I don’t see how full SWE replacement can happen unless non-SWE people using LLMs become technical enough to get what they need out of them, in which case they probably have just become SWE anyway.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#20
I believe the outcome of this type of article is actually positive. The ‘SWE-Lancer’ benchmark provides visibility into a more pragmatic assessment of LLM capabilities.

Ironically it actually refutes Altman’s claims mentioned in the same article . Hard to replace engineers when you create a benchmark you can’t score decently on.

Post reply on HN