Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

futurism.com

Post reply on HN
Print view View original

101–110 of 174 posts

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#101

Post by dkjaudyeqooe »

tkgally wrote: post #56
jasonthorsness wrote: post #19

Half of the work is specification and iteration. I think there’s a focus on full SWE replacement because it’s sensational, but we’ll more end up with SWE able to focus on the less patterned or ambiguous work and made way more productive with the LLM handling subtasks more efficiently. I don’t see how full SWE replacement can happen unless non-SWE people using LLMs become technical enough to get what they need out of…

> unless non-SWE people using LLMs become technical enough to get what they need out of them Non-SWE person here. In the past year I've been able to use LLMs to do several tasks for which I previously would have paid a freelancer on Fiverr. The most complex one, done last spring, involved writing a Python program that I ran on Google Colab to grab the OCR transcriptions of dozens of 19th-century books off the Interne…

If the goal is to get something to run correctly roughly once with some known data or input, then that's fine. Actual software development aims to run under 100% of circumstances, and LLMs are essentially cargo culting the development process and entrusting an automation that is unreliable to do mundane tasks. Sadly the quality of software will keep going down, perhaps even faster.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#102

Post by unification_fan »

pzo wrote: post #35

> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…

> How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)?

The experienced ones can

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#103

Post by blitzar »

deegles wrote: post #40
rurp wrote: post #36

I recently had to do a one-off task using SQL in a way that I wasn't too familiar with. Since I could explain conceptually what I needed but didn't know all the right syntax this seemed like a perfect use case to loop in Claude. The first couple back and forths went ok but it quickly gave me some SQL that was invalid. I sent back the exact error and line number and it responded by changing all of the aliases but repe…

I'm not convinced LLMs will evolve into general AI. The promises that it's just around the corner feels increasingly like a big scam.

We just need a little more of your (not mine) money to get there. Would I lie to you for $100 billion?

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#104

Post by anandnair »

DarkmSparks wrote: post #64
anandnair wrote: post #38

Coding, especially the type mentioned in the article (building an app based on a specification)—is a highly complex task. It cannot be completed with a single prompt and an immediate, flawless result. This is why even most software projects (built by humans) go through multiple iterations before they work perfectly. We should consider a few things before asking, "Can AI code like humans?": - How did AI learn to code?…

>How did AI learn to code? It didn't, it's just very good at copying already existing code and tweeking it a bit. >Did AI receive mentoring from an experienced senior It doesnt even comprehend what an experienced senior is, all it cares about is how frequently certain patterns occurred in certain circumstances. >Did the AI learn through hands-on coding or just by reading Stack Overflow? it "learnt" by collecting a la…

I think we're drastically oversimplifying what "pattern matching" means. It is also one of the fundamental mechanisms by which the human brain operates. I believe we are consciously (or perhaps subconsciously) conditioned to think that human "logic" and "reasoning" are several degrees more advanced than pattern matching. However, I don't think this is true.

The fundamental difference lies in how patterns are formed in each case. For LLMs, all they know are the patterns they observe in "words" - that is the only "sense" they possess. But for humans, pattern recognition involves continuously ingesting and identifying patterns across our five primary senses—not just separately, but simultaneously.

For example, when an LLM describes something as "ball-shaped," it cannot feel the shape of a ball because it lacks another sense to associate with the word "ball-shaped." In contrast, humans have the sense of touch, allowing them to associate the word or sound pattern "ball" with the physical sensation of holding a ball.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#106

Post by ilrwbwrkhv »

ginvok wrote: post #105

AI don't "solve" problems, best it can do is remember them. Ask them to solve anything new that's challenging and it starts to hallucinate. At least currently.

And I'm ashamed that OpenAI and Sam altman are walking around talking about AGI. And I'm so... disillusioned by the entire tech community that they have fallen for it or they at least pretend to believe it. It's like LinkedIn Where everybody pretends to be cringe, positivity, people. Even though they know it's cringe and nobody believes it.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#108

Post by logicallee »

rurp wrote: post #36

I recently had to do a one-off task using SQL in a way that I wasn't too familiar with. Since I could explain conceptually what I needed but didn't know all the right syntax this seemed like a perfect use case to loop in Claude. The first couple back and forths went ok but it quickly gave me some SQL that was invalid. I sent back the exact error and line number and it responded by changing all of the aliases but repe…

I am a paying user of both Claude AI and ChatGPT, I think for the use case you mention ChatGPT would have done better than Claude. At $20/month I recommend that you try it for the same use case. o1 might have succeeded where Claude failed.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#109

Post by DarkmSparks »

anandnair wrote: post #104
DarkmSparks wrote: post #64

Earlier quoted context omitted.

>How did AI learn to code? It didn't, it's just very good at copying already existing code and tweeking it a bit. >Did AI receive mentoring from an experienced senior It doesnt even comprehend what an experienced senior is, all it cares about is how frequently certain patterns occurred in certain circumstances. >Did the AI learn through hands-on coding or just by reading Stack Overflow? it "learnt" by collecting a la…

I think we're drastically oversimplifying what "pattern matching" means. It is also one of the fundamental mechanisms by which the human brain operates. I believe we are consciously (or perhaps subconsciously) conditioned to think that human "logic" and "reasoning" are several degrees more advanced than pattern matching. However, I don't think this is true. The fundamental difference lies in how patterns are formed i…

>`It is also one of the fundamental mechanisms by which the human brain operates.

One of the fundamental mechanisms by which brains operate. The bits we share with every other animal with a brain,

good luck teaching your dog to code.

being great at fetching your newspaper in the morning doesn't mean its going to wake up and write you an accounting software package at the end of the year.

Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems

#110

Post by numba888 »

krupan wrote: post #78
hackit2 wrote: post #67

Earlier quoted context omitted.

Makes perfect sense why it couldn't answer your question, you didn't have the vocabulary of relational algebra to correctly prime the model. Any rudimentary field have their own corpus vocabulary to express ideas and concepts specific to that domain.

I honestly can't tell if this is a sarcastic reply or not

I'm not sure it was even human.
Post reply on HN