Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT unexpectedly began speaking in a user's cloned voice during testing

arstechnica.com

Post reply on HN
Print view View original

81–90 of 164 posts

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#81

Post by pizzafeelsright »

c0mbonat0r wrote: post #38
skybrian wrote: post #27

It’s “unexpected” because their early training didn’t get rid of it as well as they hoped. LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. For now, it’s fairly harmless since it’s only a blooper in a lab, but there will likely be open-weights versions of this sort of thing eventually. And there will probably be p…

Can someone pleasee convince why i shouldn't be absolutely shit out of my mind cynical about this innovation? we are literally seeing the downfall of trust in society. and no, i dont believe i am exaggerating

I think you're seeing the lack of trust exposed.

Contracts allow for recompense if there is mistrust. We've been signing contracts forever.

Every automated system needs someone to jail in the event of failure.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#82

Post by dragonwriter »

Panoramix wrote: post #66
throwaway48540 wrote: post #50

Earlier quoted context omitted.

We are just finally seeing the mainstream get the idea they shouldn't trust anything on the internet. Long overdue.

Before these developments it was impossible to fake a politician saying arbitrary stuff. This is a major shift, there's no denying that.

> Before these developments it was impossible to fake a politician saying arbitrary stuff.

Voice impersonation has been possible forever, actually; and it use for misinformation (as well as less nefarious things like entertainment) is hardly novel. (Same with impersonation that goes beyond voice.)

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#83

Post by bee_rider »

throwaway48540 wrote: post #69
bloopernova wrote: post #64

Earlier quoted context omitted.

https://knowyourmeme.com/memes/torment-nexus

I know what the quote is about. I don't get how a text/audio generator is relevant to it. Or even AI - all the classic scifi I read sees AI as a positive development, it's only the mainstream action movies that present it as some catastrophe, usually in an even dumber way than the alien invasion movies. Whenever a classic scifi presented AI as part of a catastrophe, it wasn't the AI itself but some imperialist or fas…

I don’t think the popular meme is dependent on some cultivated/exclusive understanding of classic sci-fi that excludes stuff like the Terminator movies. Memes by their nature are lowest common denominator.

But hey, if you missed the joke by virtue of being too clever and well read, that’s not so bad?

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#84

Post by lolinder »

wyager wrote: post #78
lolinder wrote: post #72

Earlier quoted context omitted.

> LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. This is a great summary of almost everything that goes wrong with LLM applications. LLMs are autocomplete machines, which is why GitHub Copilot is still the most reliably useful application of LLM tech out there. The further you get from autocomplete, the less rel…

You are making a very popular mistake: confusing the training procedure for LLMs (autocomplete) with "how they work"/their internal ontology (mostly unknown). When we teach children how to do arithmetic, we have them predict missing items in equations. We don't accuse them of "only doing autocomplete". The same applies for LLMs.

I was implementing my own transformer-based models and fine tuning GPT-2 in 2019, and I've kept up with every development since then. I understand the internal structure of these things better than nearly all of the "AI Engineers" who are currently working on wrapping them up as black boxes embedded in applications.

I'm not making a "popular mistake", I'm literally describing how inference is done.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#85

Post by Animats »

sigmoid10 wrote: post #7

This problem appeared during pre-release testing and has since been solved post-generation using an output classifier that verifies responses, according to the system card release. It was predictable that someone would spin this into a black mirror-esque clickbait story.

Had to track through five links to find out what a "system card" is. It's an instance of a "model card", some broad headings to be filled in.[1]

• Model Details.

• Intended Use.

• Factors. Factors could include demographic or phenotypic groups, environmental conditions, technical attributes, or others

• Metrics.

• Evaluation Data.

• Training Data

• Quantitative Analyses

• Caveats and Recommendations

Meta's PR describes a "system card", but doesn't provide one.[2]

It doesn't, interestingly, seem to include the initial built-in prompts used as constraints.

[1] https://arxiv.org/pdf/1810.03993

[2] https://ai.meta.com/blog/system-cards-a-new-resource-for-und...

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#86

Post by lolinder »

ttul wrote: post #80
lolinder wrote: post #72

Earlier quoted context omitted.

> LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. This is a great summary of almost everything that goes wrong with LLM applications. LLMs are autocomplete machines, which is why GitHub Copilot is still the most reliably useful application of LLM tech out there. The further you get from autocomplete, the less rel…

I'd respectfully disagree with this characterization of LLMs. While they certainly excel at pattern recognition, calling them mere "autocomplete machines" vastly undersells their capabilities. LLMs demonstrate complex reasoning, multi-modal understanding, and emergent behaviors that go well beyond simple pattern continuation. They've succeeded in areas like mathematical problem-solving, creative tasks, and various re…

I never said that we couldn't use them for other things—it turns out that a lot of problems can be translated to autocomplete! And yes, in many cases the loss can be made minimal by being clever with the translation.

What I'm saying is that nearly every failure of LLM applications boils down to the engineers not understanding that they're just autocomplete engines. Most new LLM products are put together by someone who claims to be an "AI Engineer" but who approaches an LLM with magical thinking and only a vague understanding of the way they work internally.

They're autocomplete machines. That's not a bad thing or an argument that they're incapable of solving complex problems, it's just the reality of what they're doing, and a good AI engineer understands that and works with it.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#87

Post by throwaway48540 »

bee_rider wrote: post #83
throwaway48540 wrote: post #69

Earlier quoted context omitted.

I know what the quote is about. I don't get how a text/audio generator is relevant to it. Or even AI - all the classic scifi I read sees AI as a positive development, it's only the mainstream action movies that present it as some catastrophe, usually in an even dumber way than the alien invasion movies. Whenever a classic scifi presented AI as part of a catastrophe, it wasn't the AI itself but some imperialist or fas…

I don’t think the popular meme is dependent on some cultivated/exclusive understanding of classic sci-fi that excludes stuff like the Terminator movies. Memes by their nature are lowest common denominator. But hey, if you missed the joke by virtue of being too clever and well read, that’s not so bad?

Terminator is pretty good (= fun to watch) as far as action movies go, even though I have a hard time calling it scifi - there's not much science going on, except that there's a robot. But ok, the label is not really important. But using it as some sort of prediction about future is just... Wtf. At least use the scifi stories that the author actually intended to be taken as such (not many of them, and AI is usually what propels the societies leaps forward).

This is not about being well read or too clever. Many people actually think this is some sort of torment nexus and that "scifi has proven you shouldn't build it". Well a) it didn't, b) why are you taking advice from fiction designed to sell books/cinema tickets, and c) where's the joke?

There are movies about cars transforming to huge humanoid destroyers, and yet nobody claims the next Ford car model is the torment nexus. This is the same. A text/audio generator is never going to do a robot uprising, it's just as likely as your neighbor's Ford destroying your house with its giant fists.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#88

Post by AmericanChopper »

FeepingCreature wrote: post #75
AmericanChopper wrote: post #51

Earlier quoted context omitted.

> there will likely be open-weights versions of this sort of thing eventually. And there will probably be people who argue that it’s a good thing, somehow. If this technology is going to exist (which it is), do you think it would be more harmful for everybody to have access to it, or for only some group of elites (big business, your own government, somebody else government, criminal groups, whoever) to have access to…

Definitely less harmful if only the elites have it. It doesn't make them harder to kill, but it reduces proliferation. And "elites" per se isn't correlated with any political side, so it doesn't even change the culture war. Generally speaking if you think most of the danger of any technology comes from accidents, you want fewer people to have it and don't care too much which ones it is, though it'd be preferable if t…

> And "elites" per se isn't correlated with any political side, so it doesn't even change the culture war.

Even if you trust your own government, which I would suggest is misguided (the known propaganda programs that have come out of the DOD alone should be enough to dispel this trust), in a world where only the regulating authorities have access to GenAI, the GRU and the PLA are going to have access to it too. This idea that the authorities who would control GenAI wouldn’t maliciously deploy it against you is a fantasy.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#89

Post by slaveurit »

FeepingCreature wrote: post #75
AmericanChopper wrote: post #51

Earlier quoted context omitted.

> there will likely be open-weights versions of this sort of thing eventually. And there will probably be people who argue that it’s a good thing, somehow. If this technology is going to exist (which it is), do you think it would be more harmful for everybody to have access to it, or for only some group of elites (big business, your own government, somebody else government, criminal groups, whoever) to have access to…

Definitely less harmful if only the elites have it. It doesn't make them harder to kill, but it reduces proliferation. And "elites" per se isn't correlated with any political side, so it doesn't even change the culture war. Generally speaking if you think most of the danger of any technology comes from accidents, you want fewer people to have it and don't care too much which ones it is, though it'd be preferable if t…

[flagged]

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#90

Post by cdrini »

c0mbonat0r wrote: post #38
skybrian wrote: post #27

It’s “unexpected” because their early training didn’t get rid of it as well as they hoped. LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. For now, it’s fairly harmless since it’s only a blooper in a lab, but there will likely be open-weights versions of this sort of thing eventually. And there will probably be p…

Can someone pleasee convince why i shouldn't be absolutely shit out of my mind cynical about this innovation? we are literally seeing the downfall of trust in society. and no, i dont believe i am exaggerating

I think the other comments make a good argument about how other forms of technology have also degraded trust, but that we've found a way through. I'll also add that I think one potential way we could reinstate trust is through signed multimedia. Cameras/microphones/etc could sign the videos/audio they create in a way that can be used to verify that the media hasn't been doctored. Not sure if that's actually a feasible approach, but it's one possibility.
Post reply on HN