Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Security weaknesses of Copilot generated code in GitHub

arxiv.org

Post reply on HN
Print view View original

71–80 of 87 posts

Re: Security weaknesses of Copilot generated code in GitHub

#71

Post by jacobr1 »

dspillett wrote: post #59
throwaway20304 wrote: post #46

Earlier quoted context omitted.

I never thought of the I in AI as a comparison to a human of average intelligence. I always understood it means intelligence as in "capable of reasoning", regardless of whether it's "kinda dumb" or "super smart" - the same way we speak about animals not being intelligent, and are looking for "intelligent alien life" in space - the aliens might not be very smart, perhaps even totally dumb, but still intelligent. The s…

A key problem is the many different readings of the word intelligent. I wouldn't call what we currently have as "capable of reasoning" for instance, though that might not be the intent and that is instead a property of "general intelligence". Of course that has linguistic issues to as it makes general intelligence (artificial or otherwise) a subset of intelligence - i.e. more specific despite adding "general" to the…

Part of the problem is that we have very vague notions of what "reasoning" actually means. If you mean simple deductive logic ... LLMs can often perform such operations today, albeit highly inconsistently. If you mean inductive reasoning and working through a problem through first principles, then they usually fail. The state of the art their are tricks to get the system to extract out the assumptions and base knowledge and then work deductively.

But every time we have an advance in machine learning, we seem to redefine intelligent activity to be beyond that. At a certain point, what is left?

Re: Security weaknesses of Copilot generated code in GitHub

#72

Post by billyjmc »

oblio wrote: post #51
steve1977 wrote: post #40

Earlier quoted context omitted.

I guess the problem is that the term "intelligent" is ambiguous and overloaded. So in one usage someone who is kinda dumb would still be intelligent, just less intelligent than others. In another usage, we use the term to describe someone of above average intelligence (which is technically not really correct and actually not very intelligent).

> So in one usage someone who is kinda dumb would still be intelligent, just less intelligent than others. 100 IQ = 50% of the population is smarter than this. 80 IQ = not sure what the percentage is, but >50% of the population is smarter than this. If any of the tables here are to be believed: https://en.wikipedia.org/wiki/IQ_classification#Historical_I... then 80 IQ could mean 80% of the population is smarter. I'll…

From the main Wiki entry on IQ tests: “The raw score of the norming sample is usually (rank order) transformed to a normal distribution with mean 100 and standard deviation 15.”

So, you’re not too far off numerically. 80 IQ is only a -1.33 Z-score. So, 9th percentile. 91% of people score higher than 80 IQ.

Re: Security weaknesses of Copilot generated code in GitHub

#73

Post by jncfhnb »

phyzome wrote: post #70
OmarShehata wrote: post #9

Earlier quoted context omitted.

That would be kind of wild. Imagine a world where whether your system was secure was just a matter of remembering to tell the AI agent "& also make it secure" before it writes your code. (could be quite real!)

This would likely help a little bit. We've already seen LLMs improve performance on some tasks by being instructed to "think carefully" first; presumably this biases it towards parts of the training set that are higher quality. But security ultimately requires comprehension, which is not something LLMs have.

Security 100% does not require comprehension in the philosophical sense

Re: Security weaknesses of Copilot generated code in GitHub

#74

Post by empath-nirvana »

lolinder wrote: post #20
prosim wrote: post #18

Earlier quoted context omitted.

That’s no longer true. Copilot uses the same ChatGPT-3.5 model as, well, ChatGPT. If it were trained on just GitHub projects, the chat features wouldn’t work at all.

You're assuming that Copilot Chat and the regular completion are the same model. Do you have a source that says so? I'd assumed that they were two different models, since they're quite different tasks.

you don't need to use the chat to test whether it was exclusively trained on github or not.

I started typing this in my IDE in a rust project I already had open:

// the following function prints out the best soccer players of all time, which teams they played for, and why they were great as a hashmap fn soccer_greats() -> Hashmap, String)> { let mut hm = HashMap::new();

it completed with the following: hm.insert("Pele", (vec!["Santos", "New York Cosmos"], "He scored over 1000 goals in his career".to_string())); hm.insert("Maradona", (vec!["Boca Juniors", "Barcelona", "Napoli"], "He won the world cup with Argentina".to_string())); hm.insert("Messi", (vec!["Barcelona"], "He has won the Ballon d'Or 6 times".to_string())); hm.insert("Ronaldo", (vec!["Barcelona", "Inter Milan", "Real Madrid", "AC Milan", "Corinthians"], "He won the world cup with Brazil".to_string())); hm.insert("Zidane", (vec!["Juventus", "Real Madrid"], "He won the world cup with France".to_string())); }

I don't believe that information is going to be on github anywhere, but i could be mistaken.

Re: Security weaknesses of Copilot generated code in GitHub

#75

Post by [deleted] »

OmarShehata wrote: post #9
jncfhnb wrote: post #3

Did they prompt it to consider security weaknesses?

That would be kind of wild. Imagine a world where whether your system was secure was just a matter of remembering to tell the AI agent "& also make it secure" before it writes your code. (could be quite real!)

[deleted]

Re: Security weaknesses of Copilot generated code in GitHub

#76

Post by ape4 »

TheRoque wrote: post #19
crooked-v wrote: post #13

Earlier quoted context omitted.

> Copilot gives you popular responses not correct ones. That also sums up most of the issues with LLMs in general in one sentence.

Sums up the issues with democracy too, and a ton of other stuff

Democracy is a bit different. Hopefully the goal is a democratic government isn't to be intelligent but is to just make people's lives good (better?). In theory if people find their lives are getting worse then they can replace the government. But, sure, there are many examples of it not working. Such has the government making gas prices low because the people want that when its polluting the planet that the people live on.

Re: Security weaknesses of Copilot generated code in GitHub

#77

Post by throwaway20304 »

oblio wrote: post #52
throwaway20304 wrote: post #44

Earlier quoted context omitted.

Kinda dumb, but still intelligent. The sibling comment explains it well.

https://news.ycombinator.com/item?id=37776800 If by "intelligent" you're comparing with other primates, too, sure :-)

My point is, it's not a comparison at all. Intelligence is a trait, you either have it or not. We also use the same word for "how smart you are", but that measurement doesn't change anything about AI being intelligent or not. It can be dumb, but intelligent.

Re: Security weaknesses of Copilot generated code in GitHub

#79

Post by arp242 »

empath-nirvana wrote: post #74
lolinder wrote: post #20

Earlier quoted context omitted.

You're assuming that Copilot Chat and the regular completion are the same model. Do you have a source that says so? I'd assumed that they were two different models, since they're quite different tasks.

you don't need to use the chat to test whether it was exclusively trained on github or not. I started typing this in my IDE in a rust project I already had open: // the following function prints out the best soccer players of all time, which teams they played for, and why they were great as a hashmap fn soccer_greats() -> Hashmap , String)> { let mut hm = HashMap::new(); it completed with the following: hm.insert("Pe…

You need to indent code with two spaces for it to render as-is.

Re: Security weaknesses of Copilot generated code in GitHub

#80

Post by oblio »

throwaway20304 wrote: post #77
oblio wrote: post #52

Earlier quoted context omitted.

https://news.ycombinator.com/item?id=37776800 If by "intelligent" you're comparing with other primates, too, sure :-)

My point is, it's not a comparison at all. Intelligence is a trait, you either have it or not. We also use the same word for "how smart you are", but that measurement doesn't change anything about AI being intelligent or not. It can be dumb, but intelligent.

There's little value in that statement.

Intelligence is IMO an inherently comparative measure.

It's not "on/off", it's "smarter" (than a pile of rocks, than a slug, than another human).

So, yeah, you can be a dumb human but you'd be a smart chimpanzee. But we want to be comparing apples with apples in the context of this topic.

When people say "AI", everyone implicitly assumes the comparison with human intelligence. So "AI" needs to be as smart as the average human to be actual AI. Ok, AGI, if you prefer that term.

There's a reason AI is moving farther and farther away and we're creating new, finer, terms like ML, shape recognition, etc.

Post reply on HN