Live data from Hacker News

A study on robustness and reliability of large language model code generation

arxiv.org

131–140 of 229 posts

Re: A study on robustness and reliability of large language model code generation

#131
post #44
post #8

Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…

I just don’t see how it could save time. Programmers nearly universally agree that reading code is harder than writing it. So when you have chat GPT writing your code, you have to read and understand it to ensure it’s actually doing what you need it to without awkward bugs or problems. Given that it’s harder to read than write code, it seems to stand to reason that the “time savings” must be nil.

GPT can read and understand both code and the comments about 1000x faster than a human.

It's not just about the code it can generate out of a vacuum, it can also understand code for you.

I use it to generate doc-comments, check for differences between the comment and the code, or to explain code I'm not familiar with and don't understand myself.

E.g.: I don't use Python and I had to figure out what a spaghetti-code Python script actually did. I fed it to GPT 4 and started asking questions. That's way faster than learning Python first and then deciphering the script.

Re: A study on robustness and reliability of large language model code generation

#132
post #8

Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…

I have been trying to use ChatGPT to make a VueJS website (I’m a backend dev). What I have noticed with VueJS specifically is that it has little to no knowledge of the composition API, and when I ask it to help build a component it is pretty good at the scaffolding. The component will generally work; however, it is pretty useless for CSS styling. I probably should try Copilot as I bet that would work better. Also the…

How stable was the Vue composition API prior to the September 2021 training cut-off date for the OpenAI models?

Looks to me like https://blog.vuejs.org/posts/vue-2-7-naruto back-ported it in July of 2022, and the earliest snapshot of the documentation for it in Vue 3 was https://web.archive.org/web/20220210121247/https://vuejs.org... in February 2022 - so my guess is it mostly happened after that magic date.

Re: A study on robustness and reliability of large language model code generation

#133

Earlier quoted context omitted.

I've heard this from many people now but I cannot relate at all. Copilot has changed my entire strategy of programming (which I have been doing for 20 years) It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes. It is fantastic when I have to build functionality in languages I am not great at. I regularly copy and pa…

copilot? lol I can understand using chatgpt 4 but copilot??? copilot can barely autocomplete... and half the time it does it incorrectly when variables and other stuff are involved

Sounds like you would benefit from spending some time learning how to use Copilot.

Which isn't easy, because there's virtually no documentation for it.

Once you get the hang of it it's incredibly useful. I really feel it when I'm working in an editing environment without it now.

Re: A study on robustness and reliability of large language model code generation

#136
While I do see LLMs helping programmers a lot, I'm not super impressed by seeing it write what looks like a lot of boilerplate. If things become so common that it can be replicated by an LLM, it seems like we need to be abstracting it away.

Re: A study on robustness and reliability of large language model code generation

#137
post #44
post #8

Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…

I just don’t see how it could save time. Programmers nearly universally agree that reading code is harder than writing it. So when you have chat GPT writing your code, you have to read and understand it to ensure it’s actually doing what you need it to without awkward bugs or problems. Given that it’s harder to read than write code, it seems to stand to reason that the “time savings” must be nil.

Depends what you're doing. If you tend to hop between languages, software stacks, platforms etc. then you probably (as I do) spend a lot of time looking up what the options are to achieve any given task, finding and interpreting API calls, etc. Having an LLM spit out code that's in the right direction would be a huge time saver.

Re: A study on robustness and reliability of large language model code generation

#138

I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…

Which version of GPT are you using? I've found GPT4 moderately helpful, like getting help from someone with a wide but shallow experience of lots of things. > I can't easily fact check it Why not? How did you find out things about basic terminology or concepts you're not familiar about before GPT? Apply the same methods, although you just have to fact check rather than coming up with what to check, so removes a small…

>Why not? How did you find out things about basic terminology or concepts you're not familiar about before GPT?

Because if it tells me something that could just be found on google, I could have found it there first and not have to do as much fact checking. And if it tells me something I can't easily find, I can't tell if it's got some deep insight on harder to locate info, or if it's just made up.

Re: A study on robustness and reliability of large language model code generation

#139

I'm sympathetic to the viewpoint that GPT-4 is prone to mistakes when writing code. Unfortunately, the analysis in this paper is pretty bad and doesn't support that conclusion. The authors assume that for any given method under consideration, it must only occur within a particular pattern of other method calls and control flow instructions. But the templates they have chosen are clearly only applicable in certain sit…

[deleted]

Re: A study on robustness and reliability of large language model code generation

#140
post #44
post #8

Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…

I just don’t see how it could save time. Programmers nearly universally agree that reading code is harder than writing it. So when you have chat GPT writing your code, you have to read and understand it to ensure it’s actually doing what you need it to without awkward bugs or problems. Given that it’s harder to read than write code, it seems to stand to reason that the “time savings” must be nil.

GPT won't only writes code, but also documentation and unit tests. It can also read code for you and explain what is going on. Furthermore, even as an academic researcher, most of the code I write is simple boilerplate code that GPT excels at. This means I get to spend more time on the actual code that is hard and novel and that GPT cannot really help with.

Back when GitHub Copilot was first released I though very much like you and was somewhat underwhelmed by my experience with it, however since then the state of the art has improved dramatically.

Post reply on HN