OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
1–10 of 174 posts
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#2The models were restricted from accessing the internet and forced to develop their own solutions internally.
I think researchers will find that human coders are unable to solve most coding problems without access to the internet.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#3it's so much easier to learn from examples than from documentation in my opinion, documentation is, what I use when I want to know additional parameters or downsides of a functionality. I'm no coder though.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#4> even though CEO Sam Altman insists they will be able to beat "low-level" software engineers by the end of this year.
"low/high level" starts to lose its meaning to me because it gets used in opposite ways
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#5The models were restricted from accessing the internet and forced to develop their own solutions internally. I think researchers will find that human coders are unable to solve most coding problems without access to the internet.
If you're requiring human coders to write valid, compilable code, maybe. But if you're doing that, you're doing coding interviews wrong.
Any interviews I've run or been a part of have required the interviewee to demonstrate their problem-solving skills using pseudo-code.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#6This mirrors what I've seen. I've found that LLMs are most helpful in places where I have the most experience.
Maybe this is because of explicitness in prompt and preempting edge cases. Maybe it's because I know exactly what should be done. In these cases, I will still sometimes be surprised by a more complete answer then I was envisioning, a few edge cases that weren't front of mind.
But if I have _no_ idea things go wildly off course. I was doing some tricky frontend work with dynamically placed reactflow nodes and bezier curve edges. It took me easily 6 hours of bashing my head against the problem, and it was hard to stop using the assistant because of sunk cost. But I probably would have gotten more out of it and been faster if I'd just sat down and really broken down the problem for a few hours and then moved to implement.
The most tempting part of LLMs is letting them figure out design when you're in a time crunch. And the way it solves things when you understand the domain and the bottoms-up view of the work is deceptive in terms of capability.
And in this case, it's hoping that people on upwork understand their problems deeply. If they did, they probably wouldn't be posting on upwork. That's what they're trying to pay for.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#7So this is an in-house benchmarks after their undisclosed partnership with a previous benchmark company. Really hope they do not have their next model to vastly outperform on this benchmark in the coming weeks.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#8Well, OpenAI does currently have 288 job openings, including plenty of software engineers, so that says something.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#9The models were restricted from accessing the internet and forced to develop their own solutions internally. I think researchers will find that human coders are unable to solve most coding problems without access to the internet.
And you created the internet then?
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#10Despite the lack luster coding performance, AI has PROVEN its able to provide a rationale for profit taking job cuts, layoffs, reduced stock grants, and increased executive bonuses.
So it's not ALL bad news.