Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

40% of the code GitHub Copilot users check-in is AI generated and unmodified

microsoft.com

Post reply on HN
Print view View original

81–90 of 129 posts

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#81

Post by kakwa_ »

wokwokwok wrote: post #17
aliencat wrote: post #9

Earlier quoted context omitted.

unit test definitely a great use of copilot.

What are you talking about? I don't know anyone who uses it for this; specifically tests are really bad if they're subtly wrong. Maybe to scaffold the test function, but the actual test if completely useless if you don't trust it. So like... generate code and have robust tests, or write robust code... but, it's really really daft to generate tests that might hallucinate some random crap (and copilot really does somet…

In a recent personal C project (a custom archive file extractor), I have used chatgpt to generate tons of unit tests.

And honestly, I was extremely impressed. With a bit of context, it was able to generate almost correct test data for a fairly complex binary data format.

I've also leverage it heavily to:

* generate doxygen comments

* get usage examples of libs I never used

* create a whole bunch of utils functions.

And honestly for all these tedious tasks, it has done a far better job than I would ever had.

Comments were consistent in styling, the base examples were fairly good, and the utils functions were well made, specially the error handling (which I would probably have semi-consciously skipped tbh).

In fairness, I modified most of this code slightly to make it fit my project structure, or tweaked it a bit what the IA didn't quite get it right.

(it never fully understood some offset fields in the file format, but got pretty close to at times. And in fairness my naming for these offset fields was a bit questionable).

Heck, as a test, I even threw at it an RFC-like spec of the file format, and asked him to generate a python parser. The result was not 100% correct, but definitely a good start to iterate on.

In the end, this side project took me 2 weeks to implement with chatgpt probably saving around 1 week of dev. It also greatly helped improving the quality of the project (better doc, better tests).

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#82

Post by bananapub »

nathansherburn wrote: post #76
zarzavat wrote: post #73

Earlier quoted context omitted.

Copilot is way more than scaffolding boilerplate, it catches bugs as you are writing them. For example you expect: x[i] But copilot suggests: x[i+1] …and you think, and you think, and then one of two things happens. Either you satisfy yourself that the answer is indeed x[i], or you accept x[i+1] and sheepishly thank Copilot for catching a bug today. You could have copilot on in the background and never accept a sugge…

I found during my preview months ago that it would often write bugs that took me longer to correct than if I had written it myself. Has it gotten better? Also in the situation where it suggests x[i+1] and it's wrong, doesn't that annoy you? In both cases I'm sure this is just temporary while LLMs get better. But my experience was they were _just_ a little too annoying last time I used them.

> Has it gotten better?

yes

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#83

Post by bananapub »

js8 wrote: post #47
luxurytent wrote: post #5

I'm not surprised. A lot of the code I'll write in a day is trivial, calling helpers, writing test scaffolding, etc. I'm still building cool things the other 60% of time, but this job as we know it comes with a lot of repetition. (esp. if you write unit tests ;-))

The real question is, why we need to write "trivial" code? I think people have lost the plot there. We should be writing better compilers and libraries, instead of using AI that will write more code to deal with. The obsession with code generation instead of creating proper abstractions is a scourge of today's programming. It's a regression in software engineering.

> We should be writing better compilers and libraries

this is a bizarre claim to make and just completely ignores reality.

people ARE trying to make better compilers and libraries, and have been for decades. we are we are now after person-millennia of effort, and that still means the optima we're at requires writing some repetitive web template rendering code in JSX or whatever, which Copilot can automate. why didn't some homomorphic-monoid-transformer-auto-renderer win instead?

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#84

Post by js8 »

serpix wrote: post #68
js8 wrote: post #47

Earlier quoted context omitted.

The real question is, why we need to write "trivial" code? I think people have lost the plot there. We should be writing better compilers and libraries, instead of using AI that will write more code to deal with. The obsession with code generation instead of creating proper abstractions is a scourge of today's programming. It's a regression in software engineering.

Sometimes it is just the syntax of your language. We can't all be writing lisp where the syntax is just another abstraction.

> We can't all be writing lisp

Why not? Isn't it easier to learn new syntax than to figure out a correct prompting for AI? (Although actually, Haskell syntax would be something I would prefer to write in.)

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#85

Post by bananapub »

wokwokwok wrote: post #17
aliencat wrote: post #9

Earlier quoted context omitted.

unit test definitely a great use of copilot.

What are you talking about? I don't know anyone who uses it for this; specifically tests are really bad if they're subtly wrong. Maybe to scaffold the test function, but the actual test if completely useless if you don't trust it. So like... generate code and have robust tests, or write robust code... but, it's really really daft to generate tests that might hallucinate some random crap (and copilot really does somet…

> I don't know anyone who uses it for this

that would be surprising - it does a fantastic job of producing e.g. dumb unit tests. for instance, Copilot + a Go table testing template means you can churn out the simple "make a request to this url with this data, ensure I get a 200 and the response contains 'id' and not 'error'" extremely quickly. the code is trivial but tedious, so you can quickly inspect for sanity and run to ensure they pass, then commit and have them checking future changes.

> it's really really daft to generate tests that might hallucinate some random crap (and copilot really does sometimes).

variations on this comment are all over these threads, which is bizarre. hallucinating means you waste a few seconds reading the code it produced, not that you commit incorrect code.

this isn't like ChatGPT advising people to drink bleach, Copilot is a dumb tool offering an expert (you) suggested solutions for your expert consideration

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#86

Post by js8 »

never_inline wrote: post #62
js8 wrote: post #47

Earlier quoted context omitted.

The real question is, why we need to write "trivial" code? I think people have lost the plot there. We should be writing better compilers and libraries, instead of using AI that will write more code to deal with. The obsession with code generation instead of creating proper abstractions is a scourge of today's programming. It's a regression in software engineering.

I have made the same point elsewhere to AI enthusiasts friends of mine. The fact that defining the same task in natural language to an AI is easier than writing it in formal language, means the formal language lacks expressiveness [0], vocabulary, or more likely, both. But if they are both at same level of expressiveness and vocabulary, formal language wins, because it can spot and prevent ambiguity. The question is…

> How many languages allow to straightforwardly express `clamp m between 1.0 and 2.0`? You can argue this is vocabulary, or this is expressiveness.

And I think it would be a great place for AI in programming - automated refactoring. For example, it would see a code that clamps the value and could be replaced with a library function call clamp(m,1,2). (And it would also, presumably, made sure that the code is indeed the same.) So it could help you understand the vocabulary.

Or, if it sees you are doing this frequently, and it's not in the standard library, it would create the function.

But instead, this AI (more like artificial stupidity) will just suggest you the same thing you wrote because you don't know any better in your code base. That's not helping.

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#87

Post by js8 »

bananapub wrote: post #83
js8 wrote: post #47

Earlier quoted context omitted.

The real question is, why we need to write "trivial" code? I think people have lost the plot there. We should be writing better compilers and libraries, instead of using AI that will write more code to deal with. The obsession with code generation instead of creating proper abstractions is a scourge of today's programming. It's a regression in software engineering.

> We should be writing better compilers and libraries this is a bizarre claim to make and just completely ignores reality. people ARE trying to make better compilers and libraries, and have been for decades. we are we are now after person-millennia of effort, and that still means the optima we're at requires writing some repetitive web template rendering code in JSX or whatever, which Copilot can automate. why didn't…

I think the reality is that "if you build it they will come" is simply not always true, and worse is sometimes better. I don't think we are climbing to an optimum, at least unless it is culturally perceived that good abstractions are more valuable than just generating code.

The auto-renderers did actually win. There were GUI building tools at the end of 90s (like Visual Basic) that still put most web frameworks to shame. But, for various business reasons (mostly pricing, really) they were replaced with "web technology" - something open source but worse.

(And I am not dissing OSS, either. For example, Qt/KDE had some great abstractions that could serve as a basis for building applications. But the urge to reinvent the wheel has won.)

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#88

Post by circuit10 »

js8 wrote: post #47
luxurytent wrote: post #5

I'm not surprised. A lot of the code I'll write in a day is trivial, calling helpers, writing test scaffolding, etc. I'm still building cool things the other 60% of time, but this job as we know it comes with a lot of repetition. (esp. if you write unit tests ;-))

The real question is, why we need to write "trivial" code? I think people have lost the plot there. We should be writing better compilers and libraries, instead of using AI that will write more code to deal with. The obsession with code generation instead of creating proper abstractions is a scourge of today's programming. It's a regression in software engineering.

I guess one difference is that Copilot can guess things that it’s not sure about and have you check over it, but a compiler can’t just guess what you want it it would be very unreliable

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#89

Post by tasuki »

worrycue wrote: post #79
tasuki wrote: post #77

Earlier quoted context omitted.

> thinking, designing and abstracting novel ideas into code is the hard part where AI would fall short and just waste my time How can you be sure of that? I know scientists who use ChatGPT to rewrite their text to be clearer. They've also used it to pose novel interesting questions about a situation. On my side, I've found ChatGPT very helpful to explore/compare which library to use to solve my problem and show examp…

Can you trust it though? It can’t really reason - it’s just very good at guessing the right answer if there are a lot of examples. It also lies somewhat frequently. If we have to double check its output, it kind of defeats the purpose - at least part of it - as the “cognitive burden” is still on us. P.S. Not sure what you mean by “lining up” Haskell types … but don’t LSPs already give us clear information about the i…

> Can you trust it though?

Yes of course.

Libraries: I look at the libraries it suggests, see if they have documentation or GitHub activity.

Lining up types: say you have a `List (Maybe a)` and require a `Maybe (List a)`. This is a simple example, of course real life is much more complicated. ChatGPT tells you. Can you trust it? Well did it make the compile error go away?

(Sure, I can think of three ways to turn a `List (Maybe a)` into `Maybe (List a)`. Only one will be what I want, but it's easy to tell whether ChatGPT gave me the one I wanted.)

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#90

Post by enlyth »

sph wrote: post #80
ramblerman wrote: post #78

Earlier quoted context omitted.

> while thinking, designing and abstracting novel ideas into code is the hard part where AI would fall short and just waste my time. So don't use it there :) But I'm curious what kind of coding you do that you never have to do any scaffolding, refactoring, or come across any element of repetitiveness. That is where it shines imo. I find during those boring tasks it quite often just gets my intent, and I can say "yes…

AI is not as foolproof as many want us to believe. In my mind it's like giving a less experienced and error-prone engineer the code I'm working on to refactor. Perhaps it's because I've always been a lone wolf engineer, but I'd rather do it myself than giving quite an important task to someone less experienced than me, whether I'm writing something new (where I need to be at my best) or refactoring (where I still nee…

It's not foolproof but it does guess a lot of what you had exactly in your mind, just hadn't typed down yet.

If I start "enum { North", I will gladly accept the ", East, South, West }" autocomplete from Copilot, and this kind of thing, in aggregate, saves hours of your time.

Post reply on HN