Meaning 40% of the code on GitHub is insecure
GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment
11–20 of 161 posts
Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment
#12I use Copilot mostly as replacement for intellisense and macros. It helps me automating repetitive tasks. I would never trust Copilot for an algorithm or a snippet, I mean I would treat the code just like anything taken from StackOverflow or Github.
Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment
#13Meaning 40% of the code on GitHub is insecure
No. It means that when c = f(a, b), where a, and b are secure, and you have no clue what f does, it might still be the case that c is insecure.
Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment
#14I think this should be pretty much expected. I'm unfamiliar with how this network is trained, but I'm pretty sure the data ranking is not perfect. I'm guessing the ranking features are based on the repo stats, contributor stats, etc. Even "good" contributors will make rookie mistakes in certain areas. Interesting to imagine how GH will try to solve this issue.
It might be possible to learn from the change history of the projects. There's likely quite many commits which fix certain security issues, such as SQL injection problems. Maybe even with suitable metadata in the issues or commit messages.
Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment
#15Why would anyone use this in production? Just use Sourcegraph if you need help that badly.
Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment
#16I thought this should have been expected. Security starts with deep understanding. Some standards and practices can help avoid some types of problems, and some are even rather effective (like airgapping your systems), but there isn't any way to assure security in general other than truly understand what you are doing. ** I feel like Copilot is the wrong direction to optimize development. This is mostly going to help…
The real evil here is boilerplate code.
I've seen so much boilerplate in the Java or classic .NET Framework world, it's incredible. So many layers of DTOs, Request/Response Models and so on, that could be just generated. Or most of the time even removed completely (that would cost some "architects" their job though).
This is also true for a lot of Redux or Angular/NgRx applications. So much boilerplate, that you can't find the relevant code anymore.
Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment
#17But this problem is solved using GitHub's CodeQL for searching and filtering generated code. By combining Copilot with GitHub Semantic and GitHub CodeQL, you have a means of writing and generating the code you want in a secure way. This means that you no longer need the original source code that was used to train Codex. Training Codex and selling as a product in the form of Copilot steals the essence of the original source code used to train it, to build the future of programming, while paying nothing back to the original authors. Even Elon Musk was opposed to OpenAI exclusively licensing to Microsoft GPT.
https://edition.cnn.com/2020/09/27/tech/elon-musk-tesla-bill...
It's so transformative that people may allow it to circumvent licenses.
Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment
#18I thought this should have been expected. Security starts with deep understanding. Some standards and practices can help avoid some types of problems, and some are even rather effective (like airgapping your systems), but there isn't any way to assure security in general other than truly understand what you are doing. ** I feel like Copilot is the wrong direction to optimize development. This is mostly going to help…
The real evil here is boilerplate code. I've seen so much boilerplate in the Java or classic .NET Framework world, it's incredible. So many layers of DTOs, Request/Response Models and so on, that could be just generated. Or most of the time even removed completely (that would cost some "architects" their job though). This is also true for a lot of Redux or Angular/NgRx applications. So much boilerplate, that you can'…
This is because "Copy'n'paste" programming get's more and more common.
I see more and more juniors pasting code or shellcommands from StackOverflow with careless ease, without even pretending anymore that they're interested in how it actually works.
Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment
#19I've experienced this first hand: the autosuggest is scarily accurate and insidious at the same time. On numerous occasions I've auto-filled a 10-15 line suggestion that looked like it was exactly what I wanted to do, but made a very critical mistake (e.g. in a For loop, referencing the wrong array despite calling it the right name). Not really security related stuff, but head scratchers that make it harder to debug since I didn't actually write the code.
Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment
#20I thought this should have been expected. Security starts with deep understanding. Some standards and practices can help avoid some types of problems, and some are even rather effective (like airgapping your systems), but there isn't any way to assure security in general other than truly understand what you are doing. ** I feel like Copilot is the wrong direction to optimize development. This is mostly going to help…
The real evil here is boilerplate code. I've seen so much boilerplate in the Java or classic .NET Framework world, it's incredible. So many layers of DTOs, Request/Response Models and so on, that could be just generated. Or most of the time even removed completely (that would cost some "architects" their job though). This is also true for a lot of Redux or Angular/NgRx applications. So much boilerplate, that you can'…
(I have been professionally programming Java backends for the past 16 years).
Java is not the culprit here.
I think it is something that happened on the way that has something to do with J2EE and patterns craze we had a decade ago or two ago.
It doesn't help that frameworks like Spring and their documentation go out of their way to propagate these boilerplate-heavy patters.
Copying these lazy patterns is shortest, easiest way to get to working solution for a person that doesn't want to put any extra effort. And you can't get punished for doing this. Most developers don't even know there exist any other possibilities than mandatory controller calling service calling database layer and hordes of DTOs some people call "model".