Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Copilot regurgitating Quake code, including sweary comments

twitter.com

Post reply on HN
Print view View original

21–30 of 672 posts

Re: Copilot regurgitating Quake code, including sweary comments

#21

Post by ethangk »

louthy wrote: post #11
ethangk wrote: post #8

Earlier quoted context omitted.

This is a little off topic, but your framework looks really interesting! How come you opted for building a functional framework in C#, vs using F#? I couldn’t see anything in the README about what was specifically frustrating about F#? I ask because we’re looking at introducing it at my company.

I cofounded a company in 2005, the primary product is a never-ending C# web-application project. As the code-base grew to many millions of lines of code I started to see the very real problems of software engineering in the OO paradigm, and had the functional programming enlightenment moment . We started building some services in F#, but still had a massive amount of C# - and so I wanted the inertia of my team to be…

Ha, it's good to see I'm full of original thoughts.

That post in the wiki sums it up perfectly, much appreciated!

Re: Copilot regurgitating Quake code, including sweary comments

#23

Post by LeanderK »

captainmuon wrote: post #13
sydthrowaway wrote: post #10

What causes this in a net? I’m guessing the RNN gets in a catastrophic state..

I would say overfitting - the net doesn't "understand" the code in any meaningful sense. It just finds fitting examples and jumbles them a bit. Understanding would mean to have an internal representation related to the intention of the user, the expected behavior, and say the AST of the code. My pessimistic interpretation of this and many other recent AI applications is that it is a "better markov chain".

a markov chain can have an internal representation related to the intention of the user. I guess this example just got copied a lot and is therefore included multiple times in the training data, forcing the network to memorise it. Neural networks always memorise things that appear too frequent. Memorized Artifacts in an otherwise working neural network is usually seen as a "bug" (since the training allowed the network to cheat), not as a proof that the network didn't generalise.

Re: Copilot regurgitating Quake code, including sweary comments

#25

Post by lawl »

bencollier49 wrote: post #15

This does make me wonder if this is susceptible to the same form of trolling as that MS AI got. Commit a load of grossly offensive material to multiple repos, and wait for Copilot to start parroting it. I think they're going to need some human moderation.

Way better. It's susceptible to copyright trolling.

Put up repos with snippets for things people might commonly write. Preferably use javascript so you can easily "prove" it. Write a crawler that crawls and parses JS files to search for matching stuff in the AST. Now go full patent troll, eh, i mean copyright troll.

Re: Copilot regurgitating Quake code, including sweary comments

#26

Post by gruez »

bencollier49 wrote: post #15

This does make me wonder if this is susceptible to the same form of trolling as that MS AI got. Commit a load of grossly offensive material to multiple repos, and wait for Copilot to start parroting it. I think they're going to need some human moderation.

Offensive code is the least of my worries. What about vulnerable/exploitable code?

Re: Copilot regurgitating Quake code, including sweary comments

#27

Post by throwaway287391 »

klohto wrote: post #14

I’m really dumbfounded by the Copilot team decision to not exclude GPL licensed code. Why was this direction chosen? Is the inclusion of GPL really worth the risk and potential Google v. Oracle lawsuit? I’d like to know the reasoning.

Isn't it entirely possible that they did exclude GPL licensed code, but somebody somewhere has violated copyright and copy-pasted that snippet into non-GPL-licensed code that they trained on?

They could try to trace every single code snippet they train on to its "true source" and use the license for that, but that's not very well-defined, and is a lot harder, and it's never going to be 100%.

Re: Copilot regurgitating Quake code, including sweary comments

#28

Post by salawat »

sydthrowaway wrote: post #10

What causes this in a net? I’m guessing the RNN gets in a catastrophic state..

Neural nets aren't magic. You actually need quite a bit of complexity and modeling of interrelated problem spaces to get anything more than a childlike naivete or trauma savant-like mastery of one particular area with crippling deficiencies elsewhere.

Re: Copilot regurgitating Quake code, including sweary comments

#29

Post by 0-_-0 »

0-_-0 wrote: post #17

This is a very famous function [0] and likely appears multiple times in the training set (Google gives 40 hits for GitHub), which makes it more likely to be memorized by the network. [0]: https://en.wikipedia.org/wiki/Fast_inverse_square_root#Overv...

It's worth keeping in mind that what a neural network like this (just like GPT3) is doing is generating the most probable continuation based on the training dataset. Not the best continuation (whatever that means), simply the most likely one. If the training dataset has mostly bad code, the most likely continuation is likely to be bad as well. I think this is still valuable, you just have to think before accepting a suggestion (just like you have to think before writing code from scratch or copying something from Stack Overflow).

Re: Copilot regurgitating Quake code, including sweary comments

#30

Post by tyingq »

They have 4 hand picked examples on their homepage: https://copilot.github.com/

One has the issue with form encoding: https://news.ycombinator.com/item?id=27697884

The python example is using floats for currency, in an expense tracking context.

The golang one uses a word ("value") for a field name that's been a reserved word since SQL-1999. It will work in popular open source SQL databases, but I believe it would bomb in some servers if not delimited...which it is not.

The ruby one isn't outright terrible, but shows a very Americanized way to do street addresses that would probably become a problem later.

And these are the hand picked examples. This product seems like it needs some more thought. Maybe a way to comment, flag, or otherwise call out bad output?

Post reply on HN