The biggest scam in AI is Salesforce. They’re going to take some crappy model that they made or simply switch to a better open source model. Then they’re going to make a large deal with a cloud provider to spin up all these models for all their customers and then re-sell it as AI to their customers for 100x what OpenAI gets per month. And the quality will be lower.
OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
141–150 of 174 posts
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#142LLMs at this point are just a great replacement for Stack Overflow. If what you're doing has been heavily documented and you just need a primer or some skeletal sample code, the LLMs are great.
They are not creative at all, but 99% of my job is not creative either.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#143Earlier quoted context omitted.
Very few people are typist. Most people can use a keyboard, but the majority of non-technical people type at a speed which is orders of magnitude less than a professional typist. Another comment here mentions how they used colab while not being a SWE, but that is already miles ahead of what average people do with computers. There's people who have used computers for decades and wouldn't be able to do a sum in a sprea…
What’s the WPM cutoff to be considered a typist?
I guess 60-70 WPM at >95% accuracy? I have not obtained the needed certification :)
My mom went to a secretary/business assistant school in the '70s and the typing class (on a typewriter!) required using ten fingers and touch typing. The expectation was you'd be fast enough to transcribe someone dictating (they learned to use stenography for faster situations).
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#144I recently had to do a one-off task using SQL in a way that I wasn't too familiar with. Since I could explain conceptually what I needed but didn't know all the right syntax this seemed like a perfect use case to loop in Claude. The first couple back and forths went ok but it quickly gave me some SQL that was invalid. I sent back the exact error and line number and it responded by changing all of the aliases but repe…
There is no one "SQL", unfortunately. All of the major database engines have their own forks and extensions. If you didn't specify which one you were using (Microsoft SQL, Oracle, Postgres, SQLite, MySQL), then you didn't give the LLM enough information to work with, same as a junior engineer working blindly with only the information you give them.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#145I recently had to do a one-off task using SQL in a way that I wasn't too familiar with. Since I could explain conceptually what I needed but didn't know all the right syntax this seemed like a perfect use case to loop in Claude. The first couple back and forths went ok but it quickly gave me some SQL that was invalid. I sent back the exact error and line number and it responded by changing all of the aliases but repe…
I consider now that all LLMs are just for give you a rough shape, but you must rewrite it manually, doing judgments on what to keep or not.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#146I recently had to do a one-off task using SQL in a way that I wasn't too familiar with. Since I could explain conceptually what I needed but didn't know all the right syntax this seemed like a perfect use case to loop in Claude. The first couple back and forths went ok but it quickly gave me some SQL that was invalid. I sent back the exact error and line number and it responded by changing all of the aliases but repe…
There is no one "SQL", unfortunately. All of the major database engines have their own forks and extensions. If you didn't specify which one you were using (Microsoft SQL, Oracle, Postgres, SQLite, MySQL), then you didn't give the LLM enough information to work with, same as a junior engineer working blindly with only the information you give them.
I left that part out for brevity, but I told Claude the version of Postgres I was using at the start, and even specified that the mistake it produced is invalid in Postgres.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#147I recently had to do a one-off task using SQL in a way that I wasn't too familiar with. Since I could explain conceptually what I needed but didn't know all the right syntax this seemed like a perfect use case to loop in Claude. The first couple back and forths went ok but it quickly gave me some SQL that was invalid. I sent back the exact error and line number and it responded by changing all of the aliases but repe…
When claude gets in a loop like this the best thing to do is just start over in a new window.
When you line up claude on some good context and a good question it does really well. There are more specialized llms for sql I would try one of those. Claude is a generalist and for that, it's not great at everything.
It's really good at react and python -- as someone else mentioned -- that junior code is public and available.
However, random sql needs more "guiding" via the prompt. Explain more about the data and why it's wrong. Tell claude, "I think you're producing slop" and he will break out of his loop.
Good luck!
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#148Earlier quoted context omitted.
And I'm ashamed that OpenAI and Sam altman are walking around talking about AGI. And I'm so... disillusioned by the entire tech community that they have fallen for it or they at least pretend to believe it. It's like LinkedIn Where everybody pretends to be cringe, positivity, people. Even though they know it's cringe and nobody believes it.
Whenever AI hallucinates a little complex SQL or some tool / language it doesn't have much training data on, I think of AGI hype and Sam Altman's words on how AI can be used to cure cancer in near future.
Instead if they rightly just said it is an useful tool to be used by researchers to help them like a smart calculator for big data, it would be so much more honest and correct.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#149I recently had to do a one-off task using SQL in a way that I wasn't too familiar with. Since I could explain conceptually what I needed but didn't know all the right syntax this seemed like a perfect use case to loop in Claude. The first couple back and forths went ok but it quickly gave me some SQL that was invalid. I sent back the exact error and line number and it responded by changing all of the aliases but repe…
For what it’s worth, I recently wrote an SQL file that gave an error. I tried to fix it myself and searched the internet but couldn’t solve it. I pasted it into Claude and it solved the error immediately.
Re: OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems
#150I find the framing of this story quite frustrating. The purpose of new benchmarks is to gather tasks that today's LLMs can't solve comprehensively. It an AI lab built a benchmark that their models scored 100% on they would have been wasting everyone's time! Writing a story that effectively says "ha ha ha, look at OpenAI's models failing to beat the new benchemark they created!" is a complete misunderstanding of the r…
Shhh ... you're spoiling everybody's confirmation bias against LLMs. They are obviously terrible at coding, just as we have known all along, and everybody should laugh at them. Nothing to see here!