Earlier quoted context omitted.
Where are the low level CEOs vs high level CEOs? I'll bet AI could do their jobs right now. Can SOMEONE please write AI software to replace these people?
Are you bothered by the fact that software engineers might be easier to automate?
OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
71–80 of 174 posts
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#72> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…
Isn't this how interviews tend to work? So I think a good number of devs would, yes.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#73I believe it. I couldn't even get o1 or claude 3.5 to write a tampermonkey script that would turn off auto-scroll to bottom in LibreChat, even when uploading the html and javascript as context. Apparently it has to do with overflow anchor or something in React? Idk. I gave up.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#74Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#75Half of the work is specification and iteration. I think there’s a focus on full SWE replacement because it’s sensational, but we’ll more end up with SWE able to focus on the less patterned or ambiguous work and made way more productive with the LLM handling subtasks more efficiently. I don’t see how full SWE replacement can happen unless non-SWE people using LLMs become technical enough to get what they need out of…
Everyone is a typist now, so I don't think it is farfetched that everyone is a SWE in the future.
Most people can use a keyboard, but the majority of non-technical people type at a speed which is orders of magnitude less than a professional typist.
Another comment here mentions how they used colab while not being a SWE, but that is already miles ahead of what average people do with computers.
There's people who have used computers for decades and wouldn't be able to do a sum in a spreadsheet, nor know that is something spreadsheets can do.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#76Earlier quoted context omitted.
What would searching the Internet provide the models that they don’t already have? Most likely data sources such as stack overflow, documentation on the language it’s targeting, and a variety of relevant forum posts are already part of its training set. Unless someone else came along and said “here’s how to solve x problem step by step”, I don’t see how additional information past its cutoff point would help. (Perhap…
Why do programmers search for specific questions rather than always relying on their inherent knowledge? I’m a crappy hobbyist programmer but for me it is useful to see if someone has implemented exactly what I need, or debugged the problem I’m having. I don’t think it’s reasonable to expect programmers or LLMs to know everything about every library’s use in every context just from first principles.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#77Earlier quoted context omitted.
> unless non-SWE people using LLMs become technical enough to get what they need out of them Non-SWE person here. In the past year I've been able to use LLMs to do several tasks for which I previously would have paid a freelancer on Fiverr. The most complex one, done last spring, involved writing a Python program that I ran on Google Colab to grab the OCR transcriptions of dozens of 19th-century books off the Interne…
This is a good anecdote but most software engineering is not scripting. It’s getting waist (or neck) deep in a large codebase and many intricacies. That being said I’m very bullish on AI being able to handle more and more of this very soon. Cursor definitely does a great job giving us a taste of cross codebase understanding.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#78I recently had to do a one-off task using SQL in a way that I wasn't too familiar with. Since I could explain conceptually what I needed but didn't know all the right syntax this seemed like a perfect use case to loop in Claude. The first couple back and forths went ok but it quickly gave me some SQL that was invalid. I sent back the exact error and line number and it responded by changing all of the aliases but repe…
Makes perfect sense why it couldn't answer your question, you didn't have the vocabulary of relational algebra to correctly prime the model. Any rudimentary field have their own corpus vocabulary to express ideas and concepts specific to that domain.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#79Earlier quoted context omitted.
I'm not convinced LLMs will evolve into general AI. The promises that it's just around the corner feels increasingly like a big scam.
I mean it’s been a couple of years! It may or may not happen but “scam” means intentional deceit. I don’t think anyone actually knows where LLMs are going with enough certainty to use that pejorative.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#80> The models weren't allowed to access the internet How many software developers could solve most even simple programming problems (except 'Hello world') with zero shot style (you write in notepad then can compile only once and execute once) without access to internet (stackoverflow, google search, documentation), tools (terminal, debugger, linter, cli)? I think then it's not the best comparison to make any judgement…
I think most developers could do that if they trained. As someone who learned how to program before the internet, its just a different mindset and would take some time to adjust.
I am doing that now where changes take a day to make it to staging and no local environment. You roll with it.