Live data from Hacker News

GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment

theinsaneapp.com

141–150 of 161 posts

Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment

#141

Earlier quoted context omitted.

I love that we always use the average here for these justifications. We just slowly chip away and any and all excellence. 10x memes aside, we all know what it's like to work with a truly talented and productive engineer versus your everyday schmoe collecting a paycheck. It's a story as old as time, and yet here we are doing the exact big factory industrialization techniques other industries have done and that is comm…

It may be a tragedy, but I fail to see why it is a tragedy of the commons? Which resource that is a available to all is being overused? High-paying dev jobs? Those are not a commons in the sense that tragedy of the commons implies because lower-quality devs don't stand to benefit by only taking a smaller part of the job.

Here's a food analogy: everyone wants to buy the best looking apples, but then farmers are more incentivized to breed for looks than nutritious value, even though nutritious value is the superior metric.

Similarly, if everyone seeks to "dumb down" programming, you end up with a large pool of "dumbed down" programmers, which is counterproductive precisely because AI is imperfect and you need a higher level of expertise to compensate for its shortcomings. As Kernighan famously said: "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." Similarly, if one lets the AI do the thinking in their stead, what hope do they have of being able to debug it?

Ironically, though, programming already suffers from this exact problem in a very fundamental way: every tool exists to make a programmer's life easier, and consequently there are a lot of glue-code programmers. The few that actually impact the industry meaningfully (e.g. most notable software comes out of Bay Area) are very expensive because the supply of experts is limited.

Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment

#142
post #2

I thought this should have been expected. Security starts with deep understanding. Some standards and practices can help avoid some types of problems, and some are even rather effective (like airgapping your systems), but there isn't any way to assure security in general other than truly understand what you are doing. ** I feel like Copilot is the wrong direction to optimize development. This is mostly going to help…

> I feel like Copilot is the wrong direction to optimize development. This is mostly going to help people with already poor understanding of what they are doing create even more crap.

I can't wait until we start seeing Copilot Natives devs, who had it enabled from the moment they first opened VSCode at their "become an engineer in 3 months" bootcamp.

> To management (especially with wrong incentives) this seems like a perfect worker, because management usually doesn't understand the connection between lack of engagement and planning at design/development time with their later problems (or they don't feel it is them that is going to pay the price).

That's something I really want my competitors to do. Honestly it makes finding stocks to short much easier (or poaching talent...)

Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment

#143
post #54
post #39

Earlier quoted context omitted.

Just because objects are involved doesn't mean it is OOP code... OOP is completely misunderstood especially in Java community to the point where it is pretty difficult to see actually object oriented code. A "service" with a bunch of stateless functions (I am intentionally not calling them methods) is really just a library of routines and the class is used mostly for namespace purposes (to group related functions tog…

The pattern you are describing here (Service classes with static'ish methods together with data classes) is a very functional approach (modules and records). I would consider this much better than "real" OOP. The OOP "abuse" I was referring to is mostly caused by inheritance. Five or more levels of inheritance is not so uncommon in some enterprise business logic. And once you have to work with that, you arrived in he…

In java 16, they've added record objects to help with this pattern.

Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment

#144

Earlier quoted context omitted.

I disagree. 40% is not great, but unlike the masses of developers, this is a single system that can improve over time. Further, a system that can do most of the work but requires a security specialist to polish it is still a useful tool. What's important to recognize is that this is not a terribly novel concept. Unsecure code is written every day.

> unlike the masses of developers, this is a single system that can improve over time I’d say the exact opposite. Unlike this algorithm, developers can continue to learn. There will likely be future algorithms that are improvements, but this isn’t that.

Individual developers can learn, but they are replaced by new developers that have not learned. Sure, this specific instance of copilot is not the best, but it sure feels to me like people are discussing the concept of it, not the exact implementation right now.

Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment

#145
post #32

I'm using copilot now for some time and yeah it's more a toy than real help right now. The only time it really helped when I needed to create a named list of char codes. When it comes to more complex code than checking the code of copilot takes the same time as writing it. 90% of the time I needed to correct copilot. For me, tools like linters are way more helpful then. If I could only use ESLint or copilot, I would…

I think another risk with getting Copilot to start out, is that it might nudge you into a direction you wouldn't have gone into otherwise. Whether that is better or not, I suppose, it depends.

I'm working on a HTML Tokenizer in Deno/Typescript.

Copilot only helps with boilerplate code which could be handled by good intellisense.

When it tries to generate a function from the function name it fails so hard that it is more in your way then helpful.

Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment

#146
post #16

Earlier quoted context omitted.

The real evil here is boilerplate code. I've seen so much boilerplate in the Java or classic .NET Framework world, it's incredible. So many layers of DTOs, Request/Response Models and so on, that could be just generated. Or most of the time even removed completely (that would cost some "architects" their job though). This is also true for a lot of Redux or Angular/NgRx applications. So much boilerplate, that you can'…

An IDE or other tools will generate correct boilerplate code. Seems a gripe from someone that prefers hidden magical code that setups code behind their back. The evil is that someone trained an AI on random text , not even with some AST, so you have garbage in so no surprise you get garbage out. A true AI would understand that "the dev wants trough find all lines of text in a file that have this property", the AI jus…

Between explicit boilerplate vs. hidden magic, give me the boilerplate every time. At least then it is obvious (maybe painfully obvious) what code is executing. Hidden magic is the worst since code should be optimized for reading and diagnosing, not for write-time which only happens once.

Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment

#147

Earlier quoted context omitted.

I'm working on a Dart / Flutter project where most devs are coming from Java and Android backgrounds. For me, coming mostly from JavaScript, TypeScript, and Python, the amount of pointless over-engineering is very frustrating. We need to jam through every change through 10 layers now, because of "clean architecture". The team is very slow and can't implement even small changes quickly. The worst part is that I feel l…

> We need to jam through every change through 10 layers now, because of "clean architecture". I feel like this is a rather unfair comment because it doesn't sound like a situation created by "clean architecture." Granted, you probably should not try and force every detail into this architecture just like you should not rewrite a perfectly good library just because it does not fit into it nicely. But even then; drilli…

agree

i think the other thing is, theres clean architecture and then theres Clean Architecture TM where the thing is taken literally (leading to slavishly applying all the layers with lots of boilerplate, useless mocks and ridiculously coupled unit tests, over architected dependency injectors (assemblies) etc)

i was honestly surprised when i watched a series of lectures from mr uncle (bob) where he clarified a lot of things such as "use dependency injection only where it matters" and "unit tests should be replaced with integration tests after a system is finished being implemented" to "slavish following of agile "customs" is unproductive" etc etc

i think a lot of issues could be resolved if people took the time to think and listen carefully about these things and not stop at the first couple of search hits for "clean architecture"

edit:

heres the link: https://www.youtube.com/watch?v=7EmboKQH8lM&list=PLmmYSbUCWJ...

Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment

#149
post #129

Earlier quoted context omitted.

Merely hosting your code publicly seems like it wouldn't give GitHub the right to train AI models on it. You could even say it's against your terms of use. And to do it, they would have to go out of their way to find your repo on the web and clone it—unlikely. My impression (NOT A LAWYER) is that by hosting your code in a public repo on GitHub, you agree to their terms and give them the right to "read" your code incl…

It's not even very clear whether training an AI on OSS code is violation of those licenses. So unless you make your code public clearly under a proprietary license that clearly rejects such use, you can't really prevent people from doing that anyway. Just imagine, there's really nothing preventing people from scraping your blog to train their natural language processing AI or whatever, why would code be any different…

I bet if I trained an AI on some vocalist and released an album I'd get some legal mayhem. I do concede it might go differently for code, but none of these issues are crystal clear for me.

Re: GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment

#150
post #2

I thought this should have been expected. Security starts with deep understanding. Some standards and practices can help avoid some types of problems, and some are even rather effective (like airgapping your systems), but there isn't any way to assure security in general other than truly understand what you are doing. ** I feel like Copilot is the wrong direction to optimize development. This is mostly going to help…

For context, I'm a very experienced software engineer (I shipped products before most of my coworkers were born) and I've been using Copilot for 6-8 weeks while creating a challenging (and therefore fun!) new system.

> This is mostly going to help people with already poor understanding of what they are doing create even more crap.

I can see how people who haven't used it at length might come to that conclusion, but my experience with it calls the "mostly" part into question. I'm sure there will be cases of that. But as someone who deeply understands my craft, I'm finding significant benefits.

> What it does not help is the important parts of development -- defining domain of your problem, design good APIs and abstractions, understanding how everything works and fits together, understanding what your client needs, etc.

Quite the contrary! The last time a new tool helped me with those parts as much was when I moved from C++ to Python in 1997. What I experienced in my C++ -> Python transition was that an enormous chunk of my brainpower could shift from language gymnastics to the problem domain. Copilot gives me a similar feeling. It frequently suggests exactly the 1-3 lines of code I was about to type and saves me 30-60 seconds (easily 20 minutes in a full day of coding). Much better than that, it lets my focus stay on better abstractions, APIs, etc.

> Also, I feel this is going to help increase complexity by making more copies of same structures throughout the codebase.

We, as engineers, are still responsible for what we produce. Any tool needs to be used with critical thought. Of course there will be those who don't think enough. And it might even make them look better in the short term. But that will be exposed in the medium to long term - `git blame` will point to them as the authors of problematic code and not Copilot. When such problems arise (or even better, before they arise), some of us who are more experienced need to step up and mentor less experienced folks so that they develop good habits.

A small sample of areas it's helping me...

When I decide that I want to use different representations internally and externally for some data in a class, I initialize the internal member variables. Part way through typing Python's `@property` decorator, it's suggesting the name of the property and exactly how to use the member variables to generate the external representation I want. Over half the time, it's exactly what I was about to type. Maybe a quarter of the time it's not and I just don't accept the suggestion (or do a quick edit). And 5-10% of the time it suggests an approach that is better than what I was thinking. And that's in a very simple use case.

In other scenarios, it often sets up my loops just as I want them. Sometimes it picks column major when I want row major. I just keep typing and as soon as it's clear I want row major, it's suggesting that. Again, occasionally it surprises me with something better - if I just use that one function I rarely have a need for, the inner loop melts away. Why didn't I think of that? Well, now "I" did. The code I'm producing with Copilot is better than the code I would have written without because I'm thinking as I use it.

Where it really saves me time / focus is when I have some tricky calculation or API call that isn't hard, but there's a bunch of little details to get right. One I did yesterday... lookup a value in a dict, but the key needs to be mapped through another dict. Between the original key, the two dicts, and the variable receiving the result there are four variable names, plus one more for the mapped key (to spread it across two statements for readability). Before typing anything, I paused for a second to get the names straight in my head. Before I finished my thought, it suggested the lines, I looked at it for a second to make sure it was right, laughed because it was, and hit tab. It wasn't a hard task, but it helped me stay focused on the bigger picture.

Most of the time this doesn't feel at all like boilerplate. It's picking up my variable names and properly using the data structures I setup in other parts of the code. There's a big misconception that it's just pasting snippets in. It feels very different from that in real usage. Also, it rewards good naming habits. In the example above, how did it know I wanted to map the key through that dict? `key_mapping` was in the variable name. Easier for others to read later and for Copilot to read now.

The system I'm building is definitely better designed because of Copilot. Not because Copilot did any of the design, but because it freed me up to focus on the design more. It will have downsides, but in experienced hands it can be a great tool. I'm not affiliated with Microsoft / Github / OpenAI in any way. I'm just doing better work because I'm using it and doing better work makes me feel good. When the time comes, I'll pay for Copilot out of my own pocket if my company doesn't pay for it.

Post reply on HN