Live data from Hacker News

Coping with Copilot

sigarch.org

241–250 of 463 posts

Re: Coping with Copilot

#241
post #52
post #19

Earlier quoted context omitted.

That’s very easy to deal with. Oral exams or blue-book exams.

What's a blue-book exam? You can test loop-writing in an oral-exam, too. Thus your proposed solution for GPT-3 writing ethics essays is so good, that it fixes the original problem.

A blue book exam is where you go into the lecture hall and write out, long hand, in a blue book the material being tested on.

The blue book is a standard. https://www.amazon.com/BookFactory-Exam-Blue-Ruled-Format/dp...

https://www.thoughtco.com/blue-book-1856928

In the classes that I had that used blue books for exams, the exam book was free and given out by the proctor.

Re: Coping with Copilot

#242
post #70

Could you just use another teaching language? Does copilot work as well with e.g. Racket, Raku or Pascal, which all seem like decent CS101 languages to me?

Hi all - article author here. A reader of my article sent me this note, pointing me to some relevant hot-off-the-presses work (reproduced with permission): ----- A group recently evaluated the performance of Codex (and another model) on 18 programming languages: https://nuprl.github.io/MultiPL-E/ The high-order bits are: 1. The latest and greatest Codex is now twice as good on its own benchmark suite than the origina…

[deleted]

Re: Coping with Copilot

#243
Setting aside the academic implications for a moment, if you think that Copilot-like models won't be a significant part of programming in the future I'd like to point out this research from Google:

> We compare the hybrid semantic ML code completion of 10k+ Googlers (over three months across eight programming languages) to a control group and see a 6% reduction in coding iteration time (time between builds and tests) and a 7% reduction in context switches (i.e., leaving the IDE) when exposed to single-line ML completion. These results demonstrate that the combination of ML and SEs can improve developer productivity. Currently, 3% of new code (measured in characters) is now generated from accepting ML completion suggestions.

So 3% of all code at Google is now written by AI autocomplete, and Google developers who use this technology are 6% more efficient than those who don't[0].

Is it bad that students are cheating on homework with this? As bad as it was when students used Stackoverflow to cheat, I reckon. But will working with AI models be an essential skill to learn for new (and existing) programmers? About as much as getting comfortable with Stackoverflow was, I reckon.

[0]: https://ai.googleblog.com/2022/07/ml-enhanced-code-completio...

Re: Coping with Copilot

#244

I understand the problem here and I sympathize with professors in these circumstances. Learning the fundamentals will remain important. But as others have pointed out: if students are unwilling to learn, or are taking shortcuts, it will ultimately hurt them in the long run. One thing tho is that things like Copilot put the lie to the hypothesis (propagated mostly by the Google-style job interview) that intensely codi…

I code for a living and Copilot can't do the least part of my job, though it is possible it could find uses here and there around the fringes (perhaps not even then).

I'm coding audio DSP along the lines (real example here) of, "Given that the Dolby noise reduction system is deeply part of the sound of classic seventies and eighties recording, what parts of it can be generalized to make DSP processing producing the same general effects on the sound, and having done so, how should this be tweaked and adjusted to optimize for pleasurable results?"

This doesn't exist on Copilot. Damn near nothing I do for work exists on Copilot, and even if it (being an MIT-licensed public code library) gets slurped up into Copilot, Copilot has absolutely no way to determine whether a 0.8273 in the algorithm should be leaning towards 0.83 to brighten the sound of something happening, that it would do this, or why you'd want brighter rather than darker, or whether you are right in wanting this or would do better to go darker knowing that other things will happen to the listening experience.

Copilot's inability to do these things is much like Dall-E's inability to do art. There is much visualization and a striking absence of purpose or intention. With Copilot, it will do all the boring or common stuff for you. If that's your job, maybe you're not aiming at the strengths of humans, rather you are aiming at things too easy to automate…

Re: Coping with Copilot

#245

Earlier quoted context omitted.

I've been writing code for nearly as long and man, you have to try it. It's more like autocomplete on roids than something that generates Fibonacci methods automagically.

Not until there's a setting that can guarantee the automcomplete is based on verifiably license-unencumbered source that is automatically tracked in some kind of sourcemap that tells me which parts of "the code I didn't write" comes from which other project and file inside that project. Until then, copilot is a giant liability that ensures I can't use it for code that my company ends up owning, nor can I contribute c…

As a hypothetical, what would that case actually look like? I'm suing you because I have a strong belief that part of the codebase of your personal project was assembled from code I wrote and didn't license permissively, so now I'm claiming ownership?

Obviously IANAL so this is largely conjecture, but until we _actually_ see how this would play out in court, I'm leaning towards this being less of a legal issue than folks here act like. For personal projects, I'd say the likelihood of some other engineer reading your code, noticing a similarity or duplication, and dragging you to court for it is near 0.

Re: Coping with Copilot

#247

While it is true code pilot is quite amazing and can produce some very decent snippets, if the exercice you give is specific enought (using a special database schema, file format, or API), with objectives that are not common (ask the user specific questions, and draw conclusions out of it or react to events), then it cannot produce a fully working program. I use code pilot regularly now, and it does speed up my work,…

As I moved as well, I experienced three different Canadian universities' Computer Science Program. There was only one professor and course that I still remember 20 years later: CS408, Software Engineering, Professor Wortman, UofToronto. Class project was in four phases, cumulative (basic functionality for the application in phase 1, progressively additional functionality in other phases, frequently strongly interacti…

> About 250 other students signed a petition to the Dean on how this is unfair and awful and they will not put up with it.

That behaviour by the students disgusts me.

Re: Coping with Copilot

#248
post #165

Earlier quoted context omitted.

> We need some people who can do things like making those libraries. Do we? For things as simple as copilot tends to put out? Why? Do you also believe we need to keep people around that do other automated things? Plowing fields by hand? Hand compiling higher level languages (as the first lisp compiler was bootstrapped)? I mean, keep the information around. Don't go burning textbooks on subjects just because we automa…

I find this a very puzzling reply, and it may be that I misunderstood to what you are referring with the "they're" in "They're, for the most part, things we tried to put in libraries (or more ideally language standard libraries for a lot of things.)" It might refer to "the things Copilot does for you" or alternatively "the things you need to know yourself to be in industry." The thing is, regardless of which way you…

> I find this a very puzzling reply, and it may be that I misunderstood to what you are referring with the "they're" in "They're, for the most part, things we tried to put in libraries (or more ideally language standard libraries for a lot of things.)" It might refer to "the things Copilot does for you" or alternatively "the things you need to know yourself to be in industry."

I do see how that could be ambiguous. That's on me. I was referring to "the things Copilot does for you". Generally speaking trivial (or near trivial) algorithms.

> The thing is, regardless of which way you meant it, we need some people who can make the sort of libraries we need in part precisely because automation such as Copilot is no substitute

Fine. Some people might need to be able to implement and maintain libraries filled with generic algorithms, especially language maintainers. That's still a very different claim from the original "this is information that you're going to need if you actually want to go into the field". That claim implies that it's a universal requirement, whereas the reality is the vast majority probably don't need that knowledge.

Implementing these things is tedious, and results in a bunch of code that has to be maintained vs using something out of the standard library. For example, sort or max functions, which are at the intersection of what I see copilot generate a lot and intro to CS classes. Even without copilot, that's not really a skill that the average practitioner needs to have ready at all times, in fact I'd probably block a PR that implemented either of those things in my projects, it results in extra code that needs to be maintained and can be broken by a typo or something silly.

> The point about Larry Wall is that we don't get labor-saving tools without someone making an effort.

My comment about perl was sarcastic, and probably not helpful. That, also, is my bad.

Re: Coping with Copilot

#249

If what you’re asking of these students can easily be automated by a tool then the problem is in the assignment. Either don’t request it, knowing it can be automated with ease, or accept that bubble sorts are a long solved problem and move on to teaching other things. Lest anyone think I’m saying don’t teach important concepts or that such things no longer matter: there’s a whole world of principles and fundamentals…

> If what you’re asking of these students can easily be automated by a tool then the problem is in the assignment. Either don’t request it, knowing it can be automated with ease, or accept that bubble sorts are a long solved problem and move on to teaching other things.

If assignments that are easily automated are a waste of time, do you have any suggestions on how instructors should help students move from "I know literally nothing about writing code" to "I know the basics well enough to solve problems that aren't trivially generated by Copilot"? In my anecdotal experience (albeit years ago) as a graduate assistant, for students who are totally new to programming that phase often lasts throughout the typical intro to CS course and it's not unusual for it to stretch into the second level course.

Re: Coping with Copilot

#250

Earlier quoted context omitted.

someone can still mirror your stuff on github. I wonder if they should make a special open source license, that disallows use of the source code for the purpose of training something like Copilot.

If it's your creation you can make your own license. Copy and paste a license you like and add a few lines. This fact solves many "we need this lib but it's too open" problems. The author can make a special license just for you if they want.

As in this example: https://web.archive.org/web/20130203112329/http://dev.hasenj...
Post reply on HN