Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Security weaknesses of Copilot generated code in GitHub

arxiv.org

Post reply on HN
Print view View original

51–60 of 87 posts

Re: Security weaknesses of Copilot generated code in GitHub

#51

Post by oblio »

steve1977 wrote: post #40
oblio wrote: post #38

Earlier quoted context omitted.

I'm not sure about your angle here, but I thought IQ was calibrated to have 100 as the average value? So wouldn't 80 mean that someone is... kinda dumb?

I guess the problem is that the term "intelligent" is ambiguous and overloaded. So in one usage someone who is kinda dumb would still be intelligent, just less intelligent than others. In another usage, we use the term to describe someone of above average intelligence (which is technically not really correct and actually not very intelligent).

> So in one usage someone who is kinda dumb would still be intelligent, just less intelligent than others.

100 IQ = 50% of the population is smarter than this.

80 IQ = not sure what the percentage is, but >50% of the population is smarter than this.

If any of the tables here are to be believed:

https://en.wikipedia.org/wiki/IQ_classification#Historical_I...

then 80 IQ could mean 80% of the population is smarter.

I'll rephrase my "kinda dumb" to "dumb as rocks".

Re: Security weaknesses of Copilot generated code in GitHub

#52

Post by oblio »

throwaway20304 wrote: post #44
oblio wrote: post #38

Earlier quoted context omitted.

I'm not sure about your angle here, but I thought IQ was calibrated to have 100 as the average value? So wouldn't 80 mean that someone is... kinda dumb?

Kinda dumb, but still intelligent. The sibling comment explains it well.

https://news.ycombinator.com/item?id=37776800

If by "intelligent" you're comparing with other primates, too, sure :-)

Re: Security weaknesses of Copilot generated code in GitHub

#53

Post by ahoka »

crooked-v wrote: post #13
faeriechangling wrote: post #4

If a weakness is common, then of course Copilot is going to suggest it. Copilot gives you popular responses not correct ones. Yet if a weakness is common, it also means that human coders frequently make the same mistake as well. The studies results are rather unsurprising and its conclusions are oft-repeated advice. As many have said, treat copilot’s code in the same light you would treat a junior programmer’s code.

> Copilot gives you popular responses not correct ones. That also sums up most of the issues with LLMs in general in one sentence.

I propose calling it artificial non-diligence.

Re: Security weaknesses of Copilot generated code in GitHub

#54

Post by diggan »

crooked-v wrote: post #13
faeriechangling wrote: post #4

If a weakness is common, then of course Copilot is going to suggest it. Copilot gives you popular responses not correct ones. Yet if a weakness is common, it also means that human coders frequently make the same mistake as well. The studies results are rather unsurprising and its conclusions are oft-repeated advice. As many have said, treat copilot’s code in the same light you would treat a junior programmer’s code.

> Copilot gives you popular responses not correct ones. That also sums up most of the issues with LLMs in general in one sentence.

I'm not sure you can claim that the essential functionality of something is the issue with something.

The whole idea of LLMs is that they chose the most likely token based on the tokens before, and then sometimes chose less likely tokens. But it's all based on likelihoods.

Probably there is a huge education part missing from this, if people aren't aware that this is how it works, and they think that any LLM can "creatively" come up with it's own chain of tokens based on nothing.

Re: Security weaknesses of Copilot generated code in GitHub

#55

Post by worksonmine »

tedunangst wrote: post #31

There's only one weakness specifically identified that I can see. print("new user", username, password) Yeah, not best practice, but also pretty common for development if you wanted to check that everything is being passed to the correct function.

Not best practice? That's a very generous way to describe storing plaintext passwords in logs. I've seen this in the wild too but that's no excuse.

Re: Security weaknesses of Copilot generated code in GitHub

#56

Post by tmpX7dMeXU »

steve1977 wrote: post #47
throwaway20304 wrote: post #46

Earlier quoted context omitted.

I never thought of the I in AI as a comparison to a human of average intelligence. I always understood it means intelligence as in "capable of reasoning", regardless of whether it's "kinda dumb" or "super smart" - the same way we speak about animals not being intelligent, and are looking for "intelligent alien life" in space - the aliens might not be very smart, perhaps even totally dumb, but still intelligent. The s…

I guess my parent comment was lead a bit by the fact that nowadays AI is often conflated with superhuman intelligence. You're certainly correct in that even a "dumb" AI could still be intelligent. The interesting question is of course if that applies to LLMs or not. Are they actually intelligent or do they just look intelligent (and do we even have the means to answer those questions)?

This question is essentially thought-terminating in most contexts as most of not all people can’t answer it given how littler know about how humans work.

Nerds will also get hung up on this because they can’t stand the notion that any aspect of their job doesn’t require their immense intelligence.

For most contexts, “will this tool help me”’is a much more appropriate question. Anyone conflating the two is doing themselves a disservice.

Re: Security weaknesses of Copilot generated code in GitHub

#57

Post by chinathrow »

worksonmine wrote: post #55
tedunangst wrote: post #31

There's only one weakness specifically identified that I can see. print("new user", username, password) Yeah, not best practice, but also pretty common for development if you wanted to check that everything is being passed to the correct function.

Not best practice? That's a very generous way to describe storing plaintext passwords in logs. I've seen this in the wild too but that's no excuse.

> I've seen this in the wild too but that's no excuse.

See, the LLM also saw it in the wild...

Re: Security weaknesses of Copilot generated code in GitHub

#58

Post by steve1977 »

tmpX7dMeXU wrote: post #56
steve1977 wrote: post #47

Earlier quoted context omitted.

I guess my parent comment was lead a bit by the fact that nowadays AI is often conflated with superhuman intelligence. You're certainly correct in that even a "dumb" AI could still be intelligent. The interesting question is of course if that applies to LLMs or not. Are they actually intelligent or do they just look intelligent (and do we even have the means to answer those questions)?

This question is essentially thought-terminating in most contexts as most of not all people can’t answer it given how littler know about how humans work. Nerds will also get hung up on this because they can’t stand the notion that any aspect of their job doesn’t require their immense intelligence. For most contexts, “will this tool help me”’is a much more appropriate question. Anyone conflating the two is doing thems…

I don’t know if either question is more „appropriate“. One is more scientific and philosophical, the other is practical.

I mean a power drill is also helping me a lot, without being intelligent.

Re: Security weaknesses of Copilot generated code in GitHub

#59

Post by dspillett »

throwaway20304 wrote: post #46
dspillett wrote: post #45

Earlier quoted context omitted.

Less intelligent than average, given that 100 is¹ calibrated to be average. Assuming your use of the word intelligence takes the concept as a sliding scale not a boolean is/isn't which is implied by quoting IQ results. The way the “I” in AI is usually used seems to me to imply achieving average or better, so the aim is mediocre & upwards. steve1977 is agreeing with an opinion that results so far are at best “up to av…

I never thought of the I in AI as a comparison to a human of average intelligence. I always understood it means intelligence as in "capable of reasoning", regardless of whether it's "kinda dumb" or "super smart" - the same way we speak about animals not being intelligent, and are looking for "intelligent alien life" in space - the aliens might not be very smart, perhaps even totally dumb, but still intelligent. The s…

A key problem is the many different readings of the word intelligent. I wouldn't call what we currently have as "capable of reasoning" for instance, though that might not be the intent and that is instead a property of "general intelligence". Of course that has linguistic issues to as it makes general intelligence (artificial or otherwise) a subset of intelligence - i.e. more specific despite adding "general" to the name.

Re: Security weaknesses of Copilot generated code in GitHub

#60

Post by [deleted] »

lolinder wrote: post #14
diogenes4 wrote: post #11

Earlier quoted context omitted.

> Yet if a weakness is common, it also means that human coders frequently make the same mistake as well. It only means programmers commonly talk about it. This isn't the same thing as measuring incidence in production or distribution. Anyway, i'd argue the real question is "can the chatbot fix the code if requested to".

> It only means programmers commonly talk about it. This isn't the same thing as measuring incidence in production or distribution. Copilot was primarily trained on GitHub projects, not on communication between programmers. Patterns that frequently show up in Copilot output are most likely prevalent on GitHub, which is a pretty good indicator that they're common in production code.

[deleted]
Post reply on HN