People are missing the point here - it's not about writing code in one-shot. An LLM-enabled loop can generate the code and test and refine it until it works
A study on robustness and reliability of large language model code generation
41–50 of 229 posts
Re: A study on robustness and reliability of large language model code generation
#42Earlier quoted context omitted.
This is how I use it as well. I don’t trust it, but find it more useful than Google these days for zeroing in on a topic.
Particularly with Google Search getting shittier and shittier with time.
Re: A study on robustness and reliability of large language model code generation
#43I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
I work in data science and have to write a lot of repetitive stuff for parsing and cleaning data. It’s reduced my toil so dramatically in this respect, I can’t imagine going back to writing all that stuff again. I’m now much more ambitious in what I’ll experiment with as well because I know setting up the first stages of the data pipeline are going to be 10X less work than before.
Re: A study on robustness and reliability of large language model code generation
#44Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…
Programmers nearly universally agree that reading code is harder than writing it.
So when you have chat GPT writing your code, you have to read and understand it to ensure it’s actually doing what you need it to without awkward bugs or problems.
Given that it’s harder to read than write code, it seems to stand to reason that the “time savings” must be nil.
Re: A study on robustness and reliability of large language model code generation
#45I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
I have interviewed hundreds of engineers across the entire skill spectrum. I think GPT4 right now is about on-par with a mid level developer.
It makes a lot of mistakes, especially around counting, and usually can notice and fix them if they're pointed out. Humans do this all the time -- how often do you get a compiler error from something silly?
It often misapplies interfaces on the first go-around. Again, this is just like humans. We make mistakes, notice them, fix them. If you simulate this by telling GPT that its code produced an error it will often correct itself.
The absolutely killer feature of GPT4 is that it has these skills in every subject. It's fluent in kernel operations. Databases. Networking. Various UX frameworks. Any language.
It's definitely not perfect. But, if the alternative is hiring a mid-level human engineer, GPT4 is a really compelling alternative.
Re: A study on robustness and reliability of large language model code generation
#46I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
I've heard this from many people now but I cannot relate at all. Copilot has changed my entire strategy of programming (which I have been doing for 20 years) It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes. It is fantastic when I have to build functionality in languages I am not great at. I regularly copy and pa…
Re: A study on robustness and reliability of large language model code generation
#47I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
With everyone and all the news talking about it constantly inevitably people are going to start rolling their eyes.
This is bias. 48% is half way to 100%. Once it reaches 100% you don't have a job. You realize that right? It's halfway their to taking your job and you're underwhelmed. The bias is on your side.
It's typical. It's like an indie band is only popular when not many people know about it. Once everybody starts talking about it all the time it loses it's popularity. People start thinking it's not cool anymore. That's the bias you are suffering from.
Re: A study on robustness and reliability of large language model code generation
#48I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
You have to put it in context: How good is the human alternative? I have interviewed hundreds of engineers across the entire skill spectrum. I think GPT4 right now is about on-par with a mid level developer. It makes a lot of mistakes, especially around counting, and usually can notice and fix them if they're pointed out. Humans do this all the time -- how often do you get a compiler error from something silly? It of…
Re: A study on robustness and reliability of large language model code generation
#49Run the code it writes, if it gives an error, paste it into the chat and 90% of the time the LLM can fix the issue. People are missing the point here - it's not about writing code in one-shot. An LLM-enabled loop can generate the code and test and refine it until it works
People seemingly desire to zealously defend the LeetCode way of solving problems for some inane reason.
Re: A study on robustness and reliability of large language model code generation
#50Earlier quoted context omitted.
You have to put it in context: How good is the human alternative? I have interviewed hundreds of engineers across the entire skill spectrum. I think GPT4 right now is about on-par with a mid level developer. It makes a lot of mistakes, especially around counting, and usually can notice and fix them if they're pointed out. Humans do this all the time -- how often do you get a compiler error from something silly? It of…
A mid-level human engineer can iteratively fix the mistakes they start making, instead of that just being the final result. Can GPT-4 do that?
Hooking it up to automatically run the code in question and examine the output is a trivial undertaking - many folks have already done this.