Live data from Hacker News

Copilot regurgitating Quake code, including sweary comments

twitter.com

471–480 of 672 posts

Re: Copilot regurgitating Quake code, including sweary comments

#471

I hate to be the one that says this but I think it‘s true: "So you are an SWE and you take a break from work to go to Hackernews to complain that Github's Copilot, which is an AI-based solution meant to help SWEs, is utter shit and completely unusuable. And then you go back to writing AI-based solutions for some other profession. Which is totally not shit or anything.“ Can anybody put this more elegantly?

[deleted]

Re: Copilot regurgitating Quake code, including sweary comments

#472

Earlier quoted context omitted.

Nazi “concentration camps” were not actual concentration camps (a thing which long predates the Nazi camps), they were extermination camps for which “concentration camp” was a minimizing euphemism. US WWII “internment” and “relocation” centers were actual concentration camps (“relocation center” was itself a euphemism, but “internment” referred to a formal legal distinction impacting treaty obligations.)

Sure, but I don't know if I've ever heard anyone use the term "concentration camp" without qualifiers to refer to anything else than the nazi concentration camps (or something equivalent). If someone says that something is "_literally_ a concentration camp" I think that most people will think of ovens and genocide. Perhaps it's a regional thing, but that is how I interpreted it.

[deleted]

Re: Copilot regurgitating Quake code, including sweary comments

#473
post #408

Earlier quoted context omitted.

Quoted post unavailable.

Changing master to main was something Github did when they were taking heat for their contract with ICE. It was a nice bit of misdirection that cost them nothing, achieved nothing and garnered praise in some quarters. ICE, of course, runs an actual concentration camp which has a slightly more troublesome history than the word master. Language policing is to racism what recycling is to global warming - an attempt to s…

y'know it really seems like both purpose and outcome need to be closely examined here, if we're going to be emphasizing actual next to concentration camps.

what's the paradigm of a concentration camp? if we go straight for Auschwitz we'll get nowhere, how about the Boer concentration camps? Origin of the term after all.

What was the purpose? To concentrate the Boer population during a total war against them, so they couldn't supply and hide the belligerents.

What was the outcome? Tens of thousands of preventable deaths, mostly from disease. Success in the war, from the British perspective.

So, let me turn my spectacles to your example of, may I quote?

> an actual concentration camp

Which appears to be a migrant detention center. To put it succinctly, migrants who enter the country without filling out paperwork, and get caught, end up in one of these places for months-to-years while USG figures out what to do with them.

So a Boer concentration camp is filled by the British riding into a farmstead or town, kidnapping the women and children, and driving them out to a field and sticking them in a tent. A migrant detention center is filled with someone enters the United States without following the rules which govern that sort of behavior, and then, gets caught.

Where is the war?

Where is the excess death?

Ah well. I'm out of time and patience to express my contempt for your abuse of language and disrespect for the real horrors which you cheapen with this kind of facile speech.

Enjoy the 4th of July.

Re: Copilot regurgitating Quake code, including sweary comments

#474
This reply from @AzureDevOps is bizarre: "We understand. However, the way to report this issues related to Windows 11 is through our Windows Insider even from another device. Thanks in advance."

I think I'm gonna give "AI" a few more years.

https://twitter.com/AzureDevOps/status/1411018079849619458

Re: Copilot regurgitating Quake code, including sweary comments

#475
post #385

Earlier quoted context omitted.

In my testing of Copilot, the content filters only work on input , not output. Attempting to generate text from code containing "genocide" just has Copilot refuse to run. But you can still coerce Copilot to return offensive output given certain innocuous prompts.

Ahh, so it's the most pointless interpretation of the phrase "filters to block offensive words", where it is stopping the user from causing offense to the AI rather than the other way around.

They probably don't want to repeat Microsoft's incident with Tay, though they seem to have created their own incident which dooms the product if it wasn't already

Re: Copilot regurgitating Quake code, including sweary comments

#476
post #64
post #29

Earlier quoted context omitted.

It's worth keeping in mind that what a neural network like this (just like GPT3) is doing is generating the most probable continuation based on the training dataset. Not the best continuation (whatever that means), simply the most likely one. If the training dataset has mostly bad code, the most likely continuation is likely to be bad as well. I think this is still valuable, you just have to think before accepting a…

I have no idea how this or GPT3 works or how to evaluate them, but couldn't you argue that it's working as it should? You tell copilot to write a fast inverse square root, it gives you the super famous fast inverse square root. It'd be weird and bad if this didn't happen. As far as licenses go, idk. Presumably it could delete associated comments and change variable names or otherwise obscure where it's taking code fr…

Maybe I could build a robot that goes out in the city and steal cars.

As far as licenses go, idk. Presumably it could delete the number plate and repaint the car or otherwise obscure where it's taking the car from. Maybe this part is shady.

Maybe.

Re: Copilot regurgitating Quake code, including sweary comments

#477

Earlier quoted context omitted.

Sure, but I don't know if I've ever heard anyone use the term "concentration camp" without qualifiers to refer to anything else than the nazi concentration camps (or something equivalent). If someone says that something is "_literally_ a concentration camp" I think that most people will think of ovens and genocide. Perhaps it's a regional thing, but that is how I interpreted it.

It's not so much a regional as a political thing. Want it to sound worse? Use concentration camp. Want it to sound better? Use internment camp (or in some cases, re-education facility).

Or “Reserve”.

Re: Copilot regurgitating Quake code, including sweary comments

#478
post #467

Earlier quoted context omitted.

> fundamental weaknesses of a solely text-oriented approach to suggesting code. I don't think it is clear that such "fundamental weaknesses" exist. A text-based approach can get you incredibly far.

I mean, the cases where it tries to assign copyright to another person in a different year highlights that context other than the other text in the file is semantically extremely important, and not considered by this approach. Merely generating text which looks appropriate to the model given surrounding text is ... misguided? If you think about it, program synthesis is one of the few problems in which the system can…

> generating text which looks appropriate to the model given surrounding text is ... misguided?

Agreed - it represents a failure to adequately model/understand the task, but I don't think it is a "fundamental weakness" of text-based 'Chinese room' approaches.

> You gotta know the referents, not just the symbols. No one wants a half-broken Chinese Room.

"Knowing the referents" is not at all clearly defined. It's totally possible that, under the constraint of optimizing for next-word prediction, the model could develop an understanding of what the referents are.

You can't underestimate the level of complex behavior emerging from a big enough system under optimization. After all, all the crazy stuff we do - coding, art, etc. is produced by a system under evolutionary optimization pressure to make more of itself.

Re: Copilot regurgitating Quake code, including sweary comments

#479
post #369

Earlier quoted context omitted.

That's bonkers. And the beauty of it is that now someone could realistically do a GDPR Erasure request on the Neural Net. I do hope that they're able to reverse data out.

Since the information is encoded in model weights, I doubt that erasure is even possible. Only post-retrieval filtering would be an option. It only goes to show that intransparent black-box models have no place in the industry. The networks leak information left and right, because it's way too easy to just crawl the web and throw terabytes of unfiltered data at the training process.

I think the fact that there's no way to delete the data in question without throwing away the entire model is a feature...

The strategic goal of a GPDR erasure request would be to force GitHub to nuke this thing from orbit.

Re: Copilot regurgitating Quake code, including sweary comments

#480
post #371

Probably an excellent reminder that both Google and Microsoft decided to use your private emails for a training set to create Smart Reply behavior that can "write emails for you", and they swore up and down there's no way that could ever leak private information. We need legislation banning companies from ingesting data into AI training sets without explicit permission.

Makes me wonder if github are using private repos in their training data.

GitHub clearly stated they only used publicly available repos in this project. However, as many people are rightfully pointing out, those projects might still be either closed source or copylefted, and if Copilot regurgitates chunks of those projects, people who use it may be subject to infringement lawsuits in the future.
Post reply on HN