Live data from Hacker News

OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

futurism.com

81–90 of 174 posts

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#81
post #63

Earlier quoted context omitted.

Everyone is a typist now, so I don't think it is farfetched that everyone is a SWE in the future.

Very few people are typist. Most people can use a keyboard, but the majority of non-technical people type at a speed which is orders of magnitude less than a professional typist. Another comment here mentions how they used colab while not being a SWE, but that is already miles ahead of what average people do with computers. There's people who have used computers for decades and wouldn't be able to do a sum in a sprea…

What’s the WPM cutoff to be considered a typist?

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#82

I’ve got 15 years of coding experience at some of the biggest tech companies. My personal opinion is that most people have no clue how good these AI coding systems already are. If you use something like RepoPrompt, where you selectively choose which files to include in the prompt, and then also provide a clear description of what changes you want to make—along with a significant portion of the source code—a model lik…

Are you suggesting that OpenAI published a paper assessing their own models on real-world problems, but failed to properly use their own models? And/or that you know better than OpenAI scientists how to use OpenAI models most effectively?

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#84
post #49
post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

> How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? Many, there was a time when SO did not exist and people were able to solve non trivial problems. There was a time…

you miss my point about zero short style where you have only one shot to compile and execute you code. Even in old times when people programmed using punched cards it required a lot of reviews and iterations. This is the reason why scripting languages like python, ruby, php, javascript got popular because you had very fast feedback loop and do dozens of mini experiments. Majority of coding problems we have today are not algorithmic in nature.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#85
post #56

Half of the work is specification and iteration. I think there’s a focus on full SWE replacement because it’s sensational, but we’ll more end up with SWE able to focus on the less patterned or ambiguous work and made way more productive with the LLM handling subtasks more efficiently. I don’t see how full SWE replacement can happen unless non-SWE people using LLMs become technical enough to get what they need out of…

> unless non-SWE people using LLMs become technical enough to get what they need out of them Non-SWE person here. In the past year I've been able to use LLMs to do several tasks for which I previously would have paid a freelancer on Fiverr. The most complex one, done last spring, involved writing a Python program that I ran on Google Colab to grab the OCR transcriptions of dozens of 19th-century books off the Interne…

> I didn't put any full-time SWEs out of work, but I did take one job away from a Fiverr freelancer.

I think this is the nuance most miss when they think about how AI models will displace work.

Most seem to think “if it can’t fully replace a SWE then it’s not going to happen”

When in reality, it starts by lowering the threshold for someone who’s technical but not a SWE, to jump in and do the work themselves. Or it makes the job of an existing engineer more efficient. Each hour less work needed spread across many tasks that would have otherwise gone to an engineer eventually sum up to a full time worth of an engineer. If it’s a Fiverr dev you eliminated the work of, that means the Fiverr dev will eventually go after the work that’s remaining, putting supply pressure on other devs

It’s the same mistake many had about self driving cars not happening because they couldn’t handle every road. No, they just need to start with 1 road, master that, and then keep expanding to more roads. Until they can do all of SF, and then more and more cities

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#86
post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

As one who organises competitive programming contests on a regular basis for university students, I would say almost every single one.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#87
post #80
post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

You sound like someone who never used punch cards. I think most developers could do that if they trained. As someone who learned how to program before the internet, its just a different mindset and would take some time to adjust. I am doing that now where changes take a day to make it to staging and no local environment. You roll with it.

> You sound like someone who never used punch cards.

I hope HN never changes.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#88
post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

I barely ever look at StackOverflow as the quality of answers there is so poor. It was once good but the proliferation of duplicates[1] has really ruined it for me, as well as outdated answers not being replaced. Google search results are also crap.

I agree with your point, though. The "LLM" model just isn't a good fit for some tasks, in fact many tasks. It is good for creative writing, but even then only really because our standards for creative writing are pretty low. It doesn't write with any real creativity or flair in the writing. It can make things up and stay on topic. It is poor for anything where accuracy matters. It can't edit what it produces! Nobody writes things in one shot in reality, not even creative writing, but especially not code or technical writing. It needs to be able to do a whole suite of other things: move blocks of output around, rewrite chunks, expand chunks, condense chunks, check chunks against external sources or proper knowledge banks, compare chunks for internal consistency, and more. That is how we operate: at the level of functions or blocks of code, at the level of paragraphs and sentences and sections.

[1]: Yes, the opposite of the problem people here usually have with it, which is things being closed as duplicates. I think more duplicates should be deleted and redirected to a canonical answer, which is then a focus of improvement. Too often google searches give me barely answered or unanswered duplicates and I have to click around in the site to find the result Google clearly should have given me in the first place (better keyword matches, not closed, higher score, etc). I think StackOverflow do this intentionally so people have to click on more pages and see more ads.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#89
post #79
post #58

Earlier quoted context omitted.

I mean it’s been a couple of years! It may or may not happen but “scam” means intentional deceit. I don’t think anyone actually knows where LLMs are going with enough certainty to use that pejorative.

Is it intentional deceit to tell everyone it's leading to something when, as you correctly point out, nobody actually knows if it will?

when their stock price rises because of their words, yes.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#90
post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

As one who organises competitive programming contests on a regular basis for university students, I would say almost every single one.

[deleted]
Post reply on HN