Security weaknesses of Copilot generated code in GitHub

Post by Grimburger » Thu, Oct 05, 2023, 2:26 AM UTC

OmarShehata wrote: post #9

jncfhnb wrote: post #3

Did they prompt it to consider security weaknesses?

That would be kind of wild. Imagine a world where whether your system was secure was just a matter of remembering to tell the AI agent "& also make it secure" before it writes your code. (could be quite real!)

Post by lolinder » Thu, Oct 05, 2023, 3:00 AM UTC

diogenes4 wrote: post #11

faeriechangling wrote: post #4

If a weakness is common, then of course Copilot is going to suggest it. Copilot gives you popular responses not correct ones. Yet if a weakness is common, it also means that human coders frequently make the same mistake as well. The studies results are rather unsurprising and its conclusions are oft-repeated advice. As many have said, treat copilot’s code in the same light you would treat a junior programmer’s code.

> Yet if a weakness is common, it also means that human coders frequently make the same mistake as well. It only means programmers commonly talk about it. This isn't the same thing as measuring incidence in production or distribution. Anyway, i'd argue the real question is "can the chatbot fix the code if requested to".

Post by fragmede » Thu, Oct 05, 2023, 3:14 AM UTC

foota wrote: post #15

pylua wrote: post #5

Earlier quoted context omitted.

I wonder if llm are biased towards older, more insecure implementations because there is a higher volume of old code vs new code. Same thing with the data it is trained on — not all code requires all levels of refinement. Most of the data is probably around average.

This makes me wonder about training an LLM on one language and then fine tuning it for another. If you train over only, say, JavaScript, and then finetune for C, I imagine it will be quite bad at writing safe code, even if it makes the code look like C, because it didn't have to learn about freeing and such. Similarly, would it pick up patterns from one language and keep then in the other? Maybe an LLM trained on Kot…

Post by prosim » Thu, Oct 05, 2023, 3:23 AM UTC

lolinder wrote: post #14

diogenes4 wrote: post #11

Earlier quoted context omitted.

> Yet if a weakness is common, it also means that human coders frequently make the same mistake as well. It only means programmers commonly talk about it. This isn't the same thing as measuring incidence in production or distribution. Anyway, i'd argue the real question is "can the chatbot fix the code if requested to".

> It only means programmers commonly talk about it. This isn't the same thing as measuring incidence in production or distribution. Copilot was primarily trained on GitHub projects, not on communication between programmers. Patterns that frequently show up in Copilot output are most likely prevalent on GitHub, which is a pretty good indicator that they're common in production code.

Post by lolinder » Thu, Oct 05, 2023, 3:57 AM UTC

prosim wrote: post #18

lolinder wrote: post #14

Earlier quoted context omitted.

> It only means programmers commonly talk about it. This isn't the same thing as measuring incidence in production or distribution. Copilot was primarily trained on GitHub projects, not on communication between programmers. Patterns that frequently show up in Copilot output are most likely prevalent on GitHub, which is a pretty good indicator that they're common in production code.

That’s no longer true. Copilot uses the same ChatGPT-3.5 model as, well, ChatGPT. If it were trained on just GitHub projects, the chat features wouldn’t work at all.