Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A study on robustness and reliability of large language model code generation

arxiv.org

Post reply on HN
Print view View original

11–20 of 229 posts

Re: A study on robustness and reliability of large language model code generation

#11

Post by makestuff »

keyle wrote: post #8

Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…

I have been trying to use ChatGPT to make a VueJS website (I’m a backend dev). What I have noticed with VueJS specifically is that it has little to no knowledge of the composition API, and when I ask it to help build a component it is pretty good at the scaffolding. The component will generally work; however, it is pretty useless for CSS styling.

I probably should try Copilot as I bet that would work better. Also the VS Code integration would be really nice.

Re: A study on robustness and reliability of large language model code generation

#12

Post by ldhough »

jojobas wrote: post #4

Garbage in, garbage out. I wonder if it can contextualize the code in StackOverflow questions (i.e. why doesn't this work?) as bad and that in highly rated answers as good, or "code is code".

I've noticed that one of the most common failure patterns I get from GPT4 for code generation is that it incorrectly asserts something and then corrects itself in the same response.

ex: "This code `(some-fn 1 2)` does x because y. That is incorrect because abc"

I wondered if this has to do with common StackOverflow post formats.

Re: A study on robustness and reliability of large language model code generation

#13

Post by willsmith72 »

It's an interesting idea, but why is this written so weirdly?

"the users of LLM code generation services are actually the developers that are most vulnerable to these code that seems right -- They are always novice developers that are not familiar with the APIs that LLMs generate code for them"

It looks like they could've benefited from an LLM doing a pass-over. Are they saying it's only "novice developers" using LLMs?

Re: A study on robustness and reliability of large language model code generation

#14

Post by Gigachad »

I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output.

I've only found it useful for explaining basic terminology or concepts for a topic I'm not familiar with. Though this is the most dangerous use case since I'm not familiar with the answer, I can't easily fact check it.

Re: A study on robustness and reliability of large language model code generation

#15

Post by taspeotis »

I mean the landing page of GitHub Copilot just shoves text unencoded into a urlencoded body, this is nothing new...

    const response = await fetch(`http://text-processing.com/api/sentiment/`, {
      method: "POST",
      body: `text=${text}`,
      headers: {
        "Content-Type": "application/x-www-form-urlencoded",
      },
    });

Re: A study on robustness and reliability of large language model code generation

#16

Post by rany_ »

taspeotis wrote: post #15

I mean the landing page of GitHub Copilot just shoves text unencoded into a urlencoded body, this is nothing new... const response = await fetch(`http://text-processing.com/api/sentiment/`, { method: "POST", body: `text=${text}`, headers: { "Content-Type": "application/x-www-form-urlencoded", }, });

At least they can't be sued for false advertising. That's unfortunately commendable these days.

Re: A study on robustness and reliability of large language model code generation

#17

Post by jprete »

willsmith72 wrote: post #13

It's an interesting idea, but why is this written so weirdly? "the users of LLM code generation services are actually the developers that are most vulnerable to these code that seems right -- They are always novice developers that are not familiar with the APIs that LLMs generate code for them" It looks like they could've benefited from an LLM doing a pass-over. Are they saying it's only "novice developers" using LLM…

I think the author is simply weak at English, although there are some typos that really should have been corrected even in non-idiomatic English (e.g. missing spaces between words).

Really, as long as the premise, evidence, and results are clear and reproducible, the quality of the English doesn’t much matter.

Re: A study on robustness and reliability of large language model code generation

#19

Post by dools »

willsmith72 wrote: post #13

It's an interesting idea, but why is this written so weirdly? "the users of LLM code generation services are actually the developers that are most vulnerable to these code that seems right -- They are always novice developers that are not familiar with the APIs that LLMs generate code for them" It looks like they could've benefited from an LLM doing a pass-over. Are they saying it's only "novice developers" using LLM…

Yeah that statement seemed very strange to me too. I’ve been writing software for 20 years and I use ChatGPT4 everyday because it’s faster to edit and debug some code with a LLM than it is to write it all from scratch

Re: A study on robustness and reliability of large language model code generation

#20

Post by dools »

jprete wrote: post #17
willsmith72 wrote: post #13

It's an interesting idea, but why is this written so weirdly? "the users of LLM code generation services are actually the developers that are most vulnerable to these code that seems right -- They are always novice developers that are not familiar with the APIs that LLMs generate code for them" It looks like they could've benefited from an LLM doing a pass-over. Are they saying it's only "novice developers" using LLM…

I think the author is simply weak at English, although there are some typos that really should have been corrected even in non-idiomatic English (e.g. missing spaces between words). Really, as long as the premise, evidence, and results are clear and reproducible, the quality of the English doesn’t much matter.

The claim “They are always novice developers that are not familiar with the APIs that LLMs generate code for them” seems pretty outlandish to me.
Post reply on HN