In more impressive news, "38% of code generated by GPT-4 does not contain API misuses"
Seems to be already surpassing humans.
A study on robustness and reliability of large language model code generation
31–40 of 229 posts
Re: A study on robustness and reliability of large language model code generation
#32I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
Which version of GPT are you using? I've found GPT4 moderately helpful, like getting help from someone with a wide but shallow experience of lots of things. > I can't easily fact check it Why not? How did you find out things about basic terminology or concepts you're not familiar about before GPT? Apply the same methods, although you just have to fact check rather than coming up with what to check, so removes a small…
It’s a question of time investment, fact checking is slower you than actually learning the material directly. So if you need to fact check every single thing you’re better off just leaning it in the first place.
Re: A study on robustness and reliability of large language model code generation
#33Earlier quoted context omitted.
I have been trying to use ChatGPT to make a VueJS website (I’m a backend dev). What I have noticed with VueJS specifically is that it has little to no knowledge of the composition API, and when I ask it to help build a component it is pretty good at the scaffolding. The component will generally work; however, it is pretty useless for CSS styling. I probably should try Copilot as I bet that would work better. Also the…
Copilot probably wins because it holds more of _your_ code against the desired output. I find ChatGPT is really great at getting something going, think scaffolding wise. But if you enter the realm of gnarly stuff, it can get completely delirious with answers full of red herrings and sends you running in circles very quickly.
The only truly helpful use-case I’ve found is in generating some well defined text in a different form. Say - here’s a JSON object , please write a MD file describing it or turn it to X language struct/object.
Re: A study on robustness and reliability of large language model code generation
#34I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
Which version of GPT are you using? I've found GPT4 moderately helpful, like getting help from someone with a wide but shallow experience of lots of things. > I can't easily fact check it Why not? How did you find out things about basic terminology or concepts you're not familiar about before GPT? Apply the same methods, although you just have to fact check rather than coming up with what to check, so removes a small…
This is how I use it as well. I don’t trust it, but find it more useful than Google these days for zeroing in on a topic.
Re: A study on robustness and reliability of large language model code generation
#35I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
I don't relate, I'm currently designing a lower level data structure based on some papers and its been a pretty valuable rubber duck. I've only had one real instance of it being incorrect that wasted my time, higher ranked trait bounds in Rust can only be applied over lifetimes and instead it was suggesting I try to use them for non lifetime generics as well.
That said I'm not asking ChatGPT to write code that I use directly, I'm basically asking clarifying questions about the papers and other implementations I have on hand, and that's similar to how I use it in most cases.
For code generation I rely on co-pilot, which actually has the context of my codebase.
Edit: I will say this because someone else made me think of it: I find this tech all super useful on my like 20k loc and smaller side projects which are all self contained where as when I tried co pilot at work on a fragmented big corp code base I found it comparatively lackluster.
Re: A study on robustness and reliability of large language model code generation
#36Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…
[deleted]
Re: A study on robustness and reliability of large language model code generation
#37TL;DR: Treat LLMs like undergrad interns, and you won't be disappointed.
Re: A study on robustness and reliability of large language model code generation
#38Earlier quoted context omitted.
Which version of GPT are you using? I've found GPT4 moderately helpful, like getting help from someone with a wide but shallow experience of lots of things. > I can't easily fact check it Why not? How did you find out things about basic terminology or concepts you're not familiar about before GPT? Apply the same methods, although you just have to fact check rather than coming up with what to check, so removes a small…
This is how I use it as well. I don’t trust it, but find it more useful than Google these days for zeroing in on a topic.
Particularly with Google Search getting shittier and shittier with time.
Re: A study on robustness and reliability of large language model code generation
#39I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
I've heard this from many people now but I cannot relate at all. Copilot has changed my entire strategy of programming (which I have been doing for 20 years) It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes. It is fantastic when I have to build functionality in languages I am not great at. I regularly copy and pa…
I feel like using suggestions for line by line composition might lend itself to better review by programmers.
As opposed to ChatGPT which spits out dozens of lines of code based on a request and therefore requires more involved editing after the fact to understand what’s happening.
Re: A study on robustness and reliability of large language model code generation
#40Earlier quoted context omitted.
Which version of GPT are you using? I've found GPT4 moderately helpful, like getting help from someone with a wide but shallow experience of lots of things. > I can't easily fact check it Why not? How did you find out things about basic terminology or concepts you're not familiar about before GPT? Apply the same methods, although you just have to fact check rather than coming up with what to check, so removes a small…
It’s a question of time investment, fact checking is slower you than actually learning the material directly. So if you need to fact check every single thing you’re better off just leaning it in the first place.
In the original comment’s defense, sometimes you need to find where to start. For example: when writing a research paper, one might hastily cast a wide net before locating and drilling into a topic from a respected source.
I’m not well-versed in JavaScript (I’d like to be, but neither me nor my company have the resources right now to commit). When JS work comes up, I’ll often Google/ChatGPT to point me in a general direction before pulling up technical documentation on whatever comes back. I’m pretty good at learning, but sometimes it takes me a while to find out where to start.