Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Security weaknesses of Copilot generated code in GitHub

arxiv.org

Post reply on HN
Print view View original

41–50 of 87 posts

Re: Security weaknesses of Copilot generated code in GitHub

#41

Post by steve1977 »

baz00 wrote: post #37
steve1977 wrote: post #30

Earlier quoted context omitted.

Which is why the term Artificial Intelligence is really a misnomer for LLMs. Artificial Mediocracy might be more fitting.

That's the most accurate term I've heard to describe the situation. I think it could get worse though because when I've seen mediocre people work with mediocre people they generate sub-mediocre solutions through trying to be clever and failing spectacularly at it.

LLMs at the moment feel a bit like a buzwword-throwing dilbertian pointy haired boss. The use terms they heard somewhere and with some luck, they use them in the proper context... but without actually understanding them.

Edit: And yeah, I think I know what you mean. The expectation (or hope) that collaboration of mediocre people results in some above-average end by means of some magical synergy effect very rarely works in real life.

Re: Security weaknesses of Copilot generated code in GitHub

#42

Post by jddj »

tedunangst wrote: post #31

There's only one weakness specifically identified that I can see. print("new user", username, password) Yeah, not best practice, but also pretty common for development if you wanted to check that everything is being passed to the correct function.

I don't know if it still does it, but it used to be that if you did something like

  NonQueryResult StoreUser(User user) {
   var sql = "INSERT...
It would use string interpolation to fill out the properties

Re: Security weaknesses of Copilot generated code in GitHub

#43

Post by dubbel »

tedunangst wrote: post #31

There's only one weakness specifically identified that I can see. print("new user", username, password) Yeah, not best practice, but also pretty common for development if you wanted to check that everything is being passed to the correct function.

That is the CWE that they identify, but the code seems to store the apparently unhashed password in the database on top of that?

Re: Security weaknesses of Copilot generated code in GitHub

#44

Post by throwaway20304 »

oblio wrote: post #38
throwaway20304 wrote: post #36

Earlier quoted context omitted.

Do you think people with IQ below 80 are not intelligent?

I'm not sure about your angle here, but I thought IQ was calibrated to have 100 as the average value? So wouldn't 80 mean that someone is... kinda dumb?

Kinda dumb, but still intelligent. The sibling comment explains it well.

Re: Security weaknesses of Copilot generated code in GitHub

#45

Post by dspillett »

throwaway20304 wrote: post #36
steve1977 wrote: post #30

Earlier quoted context omitted.

Which is why the term Artificial Intelligence is really a misnomer for LLMs. Artificial Mediocracy might be more fitting.

Do you think people with IQ below 80 are not intelligent?

Less intelligent than average, given that 100 is¹ calibrated to be average. Assuming your use of the word intelligence takes the concept as a sliding scale not a boolean is/isn't which is implied by quoting IQ results.

The way the “I” in AI is usually used seems to me to imply achieving average or better, so the aim is mediocre & upwards. steve1977 is agreeing with an opinion that results so far are at best “up to average”, maybe not even that, so more like mediocre & downwards. The phrase Artificial Mediocracy does not seem at all unfair in this context.

The opinion that current models can at best achieve average results overall seem logical to me: that are essentially summarizing a large corpus of human output rather than having original thought. While the systems may “notice” links average humans don't due to not being able to process large amounts of data like that, bringing the average quality of their output up, they are similarly likely to latch onto bad common practise/understanding bringing it back down again. Average results are mediocre results, by definition. Not bad, but not outstanding in any way.

--

[1] supposedly, there are strong views about IQ being a flawed measure/measure in some quarters

Re: Security weaknesses of Copilot generated code in GitHub

#46

Post by throwaway20304 »

dspillett wrote: post #45
throwaway20304 wrote: post #36

Earlier quoted context omitted.

Do you think people with IQ below 80 are not intelligent?

Less intelligent than average, given that 100 is¹ calibrated to be average. Assuming your use of the word intelligence takes the concept as a sliding scale not a boolean is/isn't which is implied by quoting IQ results. The way the “I” in AI is usually used seems to me to imply achieving average or better, so the aim is mediocre & upwards. steve1977 is agreeing with an opinion that results so far are at best “up to av…

I never thought of the I in AI as a comparison to a human of average intelligence. I always understood it means intelligence as in "capable of reasoning", regardless of whether it's "kinda dumb" or "super smart" - the same way we speak about animals not being intelligent, and are looking for "intelligent alien life" in space - the aliens might not be very smart, perhaps even totally dumb, but still intelligent. The same applies for AI, perhaps it doesn't match even the least performing humans, but it's still intelligent.

Re: Security weaknesses of Copilot generated code in GitHub

#47

Post by steve1977 »

throwaway20304 wrote: post #46
dspillett wrote: post #45

Earlier quoted context omitted.

Less intelligent than average, given that 100 is¹ calibrated to be average. Assuming your use of the word intelligence takes the concept as a sliding scale not a boolean is/isn't which is implied by quoting IQ results. The way the “I” in AI is usually used seems to me to imply achieving average or better, so the aim is mediocre & upwards. steve1977 is agreeing with an opinion that results so far are at best “up to av…

I never thought of the I in AI as a comparison to a human of average intelligence. I always understood it means intelligence as in "capable of reasoning", regardless of whether it's "kinda dumb" or "super smart" - the same way we speak about animals not being intelligent, and are looking for "intelligent alien life" in space - the aliens might not be very smart, perhaps even totally dumb, but still intelligent. The s…

I guess my parent comment was lead a bit by the fact that nowadays AI is often conflated with superhuman intelligence. You're certainly correct in that even a "dumb" AI could still be intelligent.

The interesting question is of course if that applies to LLMs or not. Are they actually intelligent or do they just look intelligent (and do we even have the means to answer those questions)?

Re: Security weaknesses of Copilot generated code in GitHub

#48

Post by upwardbound »

throwaway20304 wrote: post #46
dspillett wrote: post #45

Earlier quoted context omitted.

Less intelligent than average, given that 100 is¹ calibrated to be average. Assuming your use of the word intelligence takes the concept as a sliding scale not a boolean is/isn't which is implied by quoting IQ results. The way the “I” in AI is usually used seems to me to imply achieving average or better, so the aim is mediocre & upwards. steve1977 is agreeing with an opinion that results so far are at best “up to av…

I never thought of the I in AI as a comparison to a human of average intelligence. I always understood it means intelligence as in "capable of reasoning", regardless of whether it's "kinda dumb" or "super smart" - the same way we speak about animals not being intelligent, and are looking for "intelligent alien life" in space - the aliens might not be very smart, perhaps even totally dumb, but still intelligent. The s…

I think you're correct although I'd like to point out that a lot of different animals are capable of reasoning. For example: https://www.cnet.com/tech/watch-a-wild-crow-tackle-a-complex...

Re: Security weaknesses of Copilot generated code in GitHub

#49

Post by devjab »

baq wrote: post #34
renewiltord wrote: post #33

Earlier quoted context omitted.

A junior programmer's code? This makes no sense. It's happening right in front of you. A junior programmer isn't going to write on my screen. I can just correct it right here I am currently holding the context in my head. These "security weakness" examples are print("first user registered, role set to admin", user, password) and pprint({"json":"somejunk", "classes": somefunc(user)}) Nah, this stuff I can easily spot…

The added complication is now you'll have to watch out for the junior+copilot combo, though it's a trade I personally am very willing to take.

You’ve had to watch out for “bad” programmers since the beginning of programming. Having been an external CS examiner for almost a decade now, I’m not too worried about LLMs in teaching because I’m not convinced they can do worse than what we’ve been doing this far. I do think it’s a little frightening that a lot of freshly educated Computer Scientists will have to “unlearn” a good amount of what they were taught to actually write good code, but on the flip side I work in a place where a big part of our BI related code is now written by people from social sciences because, well, they are easier to hire.

That’s how you end up with pipelines that can’t handle 1000 PDF documents, because they are simply designed not to scale beyond one or two documents. Because that’s what you get when you google program, or alternatively, when you use ChatGPT, and it’s fine… at least until it isn’t, but it’s not like you can’t already make a lucrative career “fixing” things once they stop being “good enough”. So I’m not sure things will really change.

If anything I think LLMs will be decent in the hands of both juniors and senior developers, it’s the mediocre developers who are in danger. At least with google programming they could easily tell if an SO answer or an article was from 20 years ago, that info isn’t readily available with LLMs. I fully expect to be paid well to clean up a lot of ChatGPT messes until the end of my career.

Re: Security weaknesses of Copilot generated code in GitHub

#50

Post by d-z-m »

darkerside wrote: post #8
pylua wrote: post #5

Earlier quoted context omitted.

I wonder if llm are biased towards older, more insecure implementations because there is a higher volume of old code vs new code. Same thing with the data it is trained on — not all code requires all levels of refinement. Most of the data is probably around average.

> Most of the data is probably around average. I know this is not how distributions work, but I had to chuckle at the literal interpretation of this.

I'd say the data is pretty normal.
Post reply on HN