Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

40% of the code GitHub Copilot users check-in is AI generated and unmodified

microsoft.com

Post reply on HN
Print view View original

21–30 of 129 posts

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#21

Post by dandellion »

wokwokwok wrote: post #17
aliencat wrote: post #9

Earlier quoted context omitted.

unit test definitely a great use of copilot.

What are you talking about? I don't know anyone who uses it for this; specifically tests are really bad if they're subtly wrong. Maybe to scaffold the test function, but the actual test if completely useless if you don't trust it. So like... generate code and have robust tests, or write robust code... but, it's really really daft to generate tests that might hallucinate some random crap (and copilot really does somet…

In my case about 50% of the test code is just boilerplate, so I usually type the test name and generate the rest. Most of the time I have to rewrite the actual test logic (although very trivial tests are sometimes correct), but I probably keep more than 40% if we're talking tests specifically.

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#23

Post by shmoogy »

halfjoking wrote: post #14
pirate787 wrote: post #4

Earlier quoted context omitted.

This is my experience as well. For longer code, Copilot introduces errors through making up variables and method names that don't actually exist on the object. These errors are very annoying but only 5% of the generated code.

I tried Copilot last month and canceled my subscription because of all the small errors in generated functions. It was more painful to debug than write it myself. Maybe I should have lowered expectations and thought of it as improved code complete. I got spoiled by copy/pasting from ChatGPT. As it stands now I'd rather pay for a subscription to have the latest ChatGPT than for Copilot.

I feel like it has been getting worse for me in the past few months. I will write a comment and it will autocomplete - but have something wrong - almost like using chatGPT to try to generate code. Mostly right - but not quite. Things it would have gotten in the past.

Maybe it's just me.

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#24

Post by Waterluvian »

rjh29 wrote: post #20
wokwokwok wrote: post #17

Earlier quoted context omitted.

What are you talking about? I don't know anyone who uses it for this; specifically tests are really bad if they're subtly wrong. Maybe to scaffold the test function, but the actual test if completely useless if you don't trust it. So like... generate code and have robust tests, or write robust code... but, it's really really daft to generate tests that might hallucinate some random crap (and copilot really does somet…

Most of the time I know exactly what tests I want to write but it's annoying to write them. Copilot autocompletes, I check it and then I use it.

For me the worst part about this is that writing tests tediously makes me reach for layers of abstraction on my test code. And then suddenly my test code is complicated and needs its own test code, and changing a test can often be problematic to the abstractions I foolishly employed.

Being able to churn out the boilerplate for tests, which can make DRY a non-concern, is great. I just hope that if I do this, I never get sloppy, and I always review the tests.

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#25

Post by teaearlgraycold »

wokwokwok wrote: post #17
aliencat wrote: post #9

Earlier quoted context omitted.

unit test definitely a great use of copilot.

What are you talking about? I don't know anyone who uses it for this; specifically tests are really bad if they're subtly wrong. Maybe to scaffold the test function, but the actual test if completely useless if you don't trust it. So like... generate code and have robust tests, or write robust code... but, it's really really daft to generate tests that might hallucinate some random crap (and copilot really does somet…

I use it for unit tests. That's one of its best use cases. Obviously I read the code it writes down.

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#26

Post by r3dneck »

IMHO, if you really "tell a CEO I can give him 56% more developers overnight" , he will sue you. Every programmer tried it out can tell that his affirmation could be right only for trivial part of a program, for non trivial programming part, it is more like a board game, where you try to induce the "AI" to produce the right answer modifying the "prompt" again, again, again, again, ( Did I tell you "again" ?) and again. Obviously that is a customer fault because he didn't hire prompt engineers to do properly the job to describe properly the job they need be done

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#27

Post by wongarsu »

aliencat wrote: post #9
luxurytent wrote: post #5

I'm not surprised. A lot of the code I'll write in a day is trivial, calling helpers, writing test scaffolding, etc. I'm still building cool things the other 60% of time, but this job as we know it comes with a lot of repetition. (esp. if you write unit tests ;-))

unit test definitely a great use of copilot.

I guess this could be true in some languages and settings, for example when a notable portion of your tests verify correct behavior with null values, arguments of wrong type, etc. When talking about unit tests that verify the actual logic, you should naturally become more careful with your copilot suggestions. Especially if copilot also wrote the code it's testing.

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#28

Post by monero-xmr »

There is a lot of annoyance in the boilerplate of a program - "set up a scaffold for this, template that, convert data from this format to that format". The AI tools I have experimented with are good at this and for sure this is useful.

But no one can read my mind! If I have to get increasingly specific in what I am telling the AI, and have to formulate my writing style in a way that keeps things very precise, I am essentially programming in another language. Albeit one where the output is uncertain and I have to check everything.

It is possible that someday it reads in an entire codebase (say, millions of lines of code), and I can describe a new API I need and it implements the entire thing. This would be quite useful.

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#29

Post by paro_nej »

dandellion wrote: post #21
wokwokwok wrote: post #17

Earlier quoted context omitted.

What are you talking about? I don't know anyone who uses it for this; specifically tests are really bad if they're subtly wrong. Maybe to scaffold the test function, but the actual test if completely useless if you don't trust it. So like... generate code and have robust tests, or write robust code... but, it's really really daft to generate tests that might hallucinate some random crap (and copilot really does somet…

In my case about 50% of the test code is just boilerplate, so I usually type the test name and generate the rest. Most of the time I have to rewrite the actual test logic (although very trivial tests are sometimes correct), but I probably keep more than 40% if we're talking tests specifically.

> In my case about 50% of the test code is just boilerplate

Why not replace it with a function call?

Re: 40% of the code GitHub Copilot users check-in is AI generated and unmodified

#30

Post by rabuse »

micromacrofoot wrote: post #6

for me it’s a fancy autocomplete at its worst! it’s a handy tool to cut down on repetitive tasks

This is exactly how I use it also. It's great when it suggests the correct thing, but doesn't interfere with me if it's wrong.
Post reply on HN