Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Security weaknesses of Copilot generated code in GitHub

arxiv.org

Post reply on HN
Print view View original

61–70 of 87 posts

Re: Security weaknesses of Copilot generated code in GitHub

#61

Post by famouswaffles »

steve1977 wrote: post #47
throwaway20304 wrote: post #46

Earlier quoted context omitted.

I never thought of the I in AI as a comparison to a human of average intelligence. I always understood it means intelligence as in "capable of reasoning", regardless of whether it's "kinda dumb" or "super smart" - the same way we speak about animals not being intelligent, and are looking for "intelligent alien life" in space - the aliens might not be very smart, perhaps even totally dumb, but still intelligent. The s…

I guess my parent comment was lead a bit by the fact that nowadays AI is often conflated with superhuman intelligence. You're certainly correct in that even a "dumb" AI could still be intelligent. The interesting question is of course if that applies to LLMs or not. Are they actually intelligent or do they just look intelligent (and do we even have the means to answer those questions)?

>The interesting question is of course if that applies to LLMs or not. Are they actually intelligent or do they just look intelligent (and do we even have the means to answer those questions)?

It's not an interesting question. It's pretty meaningless.

Are birds really flying or do they look like they are flying (perspective of the bee)?. Are planes really flying or do they look like they're flying ?

"Mimic Intelligence" is not a real distinction.

Re: Security weaknesses of Copilot generated code in GitHub

#62

Post by dangerface »

throwaway20304 wrote: post #46
dspillett wrote: post #45

Earlier quoted context omitted.

Less intelligent than average, given that 100 is¹ calibrated to be average. Assuming your use of the word intelligence takes the concept as a sliding scale not a boolean is/isn't which is implied by quoting IQ results. The way the “I” in AI is usually used seems to me to imply achieving average or better, so the aim is mediocre & upwards. steve1977 is agreeing with an opinion that results so far are at best “up to av…

I never thought of the I in AI as a comparison to a human of average intelligence. I always understood it means intelligence as in "capable of reasoning", regardless of whether it's "kinda dumb" or "super smart" - the same way we speak about animals not being intelligent, and are looking for "intelligent alien life" in space - the aliens might not be very smart, perhaps even totally dumb, but still intelligent. The s…

Humans are the only thing capable of reasoning, AI isn't capable of it and its very rare that an animal other than a human is capable of even the most basic reasoning.

Animals act on instinct that is hard coded based on the probability of survival AI essentially does the same thing it follows hardcoded probabilities not reason.

Re: Security weaknesses of Copilot generated code in GitHub

#63

Post by xmcqdpt2 »

pylua wrote: post #7
squigz wrote: post #6

Earlier quoted context omitted.

I'm not sure that code being newer inherently means it will be more secure

I don’t think it is a tautology , but I can imagine a cve scanner picking up older code with log4j where newer code may avoid that library altogether, just as an example. Since there is more older code than newer code would the llm be suspectible to that ?

Everyone still uses log4j after the fix. There aren't many full-featured alternatives... and the one that exist probably contain unfixed bugs.

Re: Security weaknesses of Copilot generated code in GitHub

#64

Post by hiAndrewQuinn »

faeriechangling wrote: post #4

If a weakness is common, then of course Copilot is going to suggest it. Copilot gives you popular responses not correct ones. Yet if a weakness is common, it also means that human coders frequently make the same mistake as well. The studies results are rather unsurprising and its conclusions are oft-repeated advice. As many have said, treat copilot’s code in the same light you would treat a junior programmer’s code.

It would be very interesting to fine-tune Copilot on the code of people widely regarded in their communities as experts, to see how the suggestions would change.

Re: Security weaknesses of Copilot generated code in GitHub

#65

Post by reportgunner »

faeriechangling wrote: post #4

If a weakness is common, then of course Copilot is going to suggest it. Copilot gives you popular responses not correct ones. Yet if a weakness is common, it also means that human coders frequently make the same mistake as well. The studies results are rather unsurprising and its conclusions are oft-repeated advice. As many have said, treat copilot’s code in the same light you would treat a junior programmer’s code.

Brawndo is great for plants because it has elecrolytes.

Re: Security weaknesses of Copilot generated code in GitHub

#66

Post by danielvaughn »

steve1977 wrote: post #30
crooked-v wrote: post #13

Earlier quoted context omitted.

> Copilot gives you popular responses not correct ones. That also sums up most of the issues with LLMs in general in one sentence.

Which is why the term Artificial Intelligence is really a misnomer for LLMs. Artificial Mediocracy might be more fitting.

Artificial Average

Re: Security weaknesses of Copilot generated code in GitHub

#67

Post by jncfhnb »

OmarShehata wrote: post #9
jncfhnb wrote: post #3

Did they prompt it to consider security weaknesses?

That would be kind of wild. Imagine a world where whether your system was secure was just a matter of remembering to tell the AI agent "& also make it secure" before it writes your code. (could be quite real!)

I think it’s more likely that you would use a security graded bot.

It’s perfectly reasonable to not use secure code for a large number of use cases.

Re: Security weaknesses of Copilot generated code in GitHub

#68

Post by Zambyte »

steve1977 wrote: post #30
crooked-v wrote: post #13

Earlier quoted context omitted.

> Copilot gives you popular responses not correct ones. That also sums up most of the issues with LLMs in general in one sentence.

Which is why the term Artificial Intelligence is really a misnomer for LLMs. Artificial Mediocracy might be more fitting.

It is artificial intelligence, it just isn't artificial general intelligence, nor artificial general knowledge. LLMs are artificial linguistic intelligence. They are really good at linguistic operations that require intelligence, like summarizing long text, transforming disrespectful text into professional looking text, translation between languages to a certain degree, etc.

It is not possible to ask an LLM for factual knowledge, without providing it the source of the fact. Without a source of the fact, you can only ask an LLM to generate an answer to the question that is linguistically convincing. And they can do a really good job at that. They can accidentally encode factual knowledge by predicting the next word correctly, but that should be regarded as an accident.

Re: Security weaknesses of Copilot generated code in GitHub

#69

Post by jacobr1 »

steve1977 wrote: post #30
crooked-v wrote: post #13

Earlier quoted context omitted.

> Copilot gives you popular responses not correct ones. That also sums up most of the issues with LLMs in general in one sentence.

Which is why the term Artificial Intelligence is really a misnomer for LLMs. Artificial Mediocracy might be more fitting.

A sci-fi series I read on occasion uses term "artificial stupids"

Re: Security weaknesses of Copilot generated code in GitHub

#70

Post by phyzome »

OmarShehata wrote: post #9
jncfhnb wrote: post #3

Did they prompt it to consider security weaknesses?

That would be kind of wild. Imagine a world where whether your system was secure was just a matter of remembering to tell the AI agent "& also make it secure" before it writes your code. (could be quite real!)

This would likely help a little bit. We've already seen LLMs improve performance on some tasks by being instructed to "think carefully" first; presumably this biases it towards parts of the training set that are higher quality.

But security ultimately requires comprehension, which is not something LLMs have.

Post reply on HN