Live data from Hacker News

ChatGPT unexpectedly began speaking in a user's cloned voice during testing

arstechnica.com

131–140 of 164 posts

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#131

Earlier quoted context omitted.

> LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. This is a great summary of almost everything that goes wrong with LLM applications. LLMs are autocomplete machines, which is why GitHub Copilot is still the most reliably useful application of LLM tech out there. The further you get from autocomplete, the less rel…

You and I are just autocomplete machines.

I’m surprised ur the first comment I e seen where a human admits humans aren’t that complex in the big scheme of things

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#132

Earlier quoted context omitted.

I don't understand how anybody can still claim LLMs show "complex reasoning". It's been shown time and time again that they'll produce a correct chain of reasoning when given a problem (e.g. wolf, goat, cabbage crossing a river; 3 guards and a door; etc.) that is roughly similar to what's in the training data but will fail when given a sufficiently novel modification _while still producing output that is confidently…

>there was simply text that is statistically likely to be arranged in that way This is wrong. An LLM can produce text that has never been arranged that way in training data.

Your statement doesn't contradict their statement. It produces text that is statistically likely to be arranged that way, but because we use non-deterministic sampling we get a wide variety of results that weren't necessarily in the training data.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#133

Earlier quoted context omitted.

In other words, would it be fair to say that any naturally sequential problem is not very far from autocomplete? Again, I think you're putting words in my mouth and thoughts in my head that aren't there. A lot of people have reacted to AI hype by going the other way and underestimating them—that's not me. I think there are lots of problems they can solve, I just think they all boil down to autocomplete and if you can…

People fine tune LLMs for classification tasks. This is completely wrong.

You're just going around saying people are completely wrong without reading what they wrote or providing any justification for that claim. I'm not sure how to respond because your comment is a non sequitur.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#134
post #15

Earlier quoted context omitted.

Did anybody with a passing familiarity with the technology not know this? There's a reason people don't trust their IP to public chat assistants, ya know. It's because they're remorselessly siphoning our data for training purposes. And you've agreed to let them use your inputs for anything they please.

That’s not what is happening here. Instead, the user’s voice output during the session is used to generate later sound by the model accidentally.

No, I get that. But that's the thing about LLMs, isn't it? They're good at filling in the next token to make a plausible-looking conversation. And if the machine doesn't expect you to be done talking, why would it not just carry on in your voice?

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#135
post #13

Earlier quoted context omitted.

Anyone can clone a voice with access to a couple of seconds of it.

We'll have to gradually get used to the notion that a person's voice, like so many other things we once thought of as intimately personal, is just a coordinate in a high-dimensional vector space.

It always has been. The same way passwords were never secure, we just found that they are too slow to bother uncovering. The same can be said for voices/AI. Except now it's not too slow to "crack an easy password" in the voices/AI world.

The idea that everything is open, you just need the right tool, is scary to most.. but it's not once you realise how expensive the tools are.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#137

Earlier quoted context omitted.

every single voice cloning project I'm sure you could have used google https://elevenlabs.io/app/voice-lab https://app.resemble.ai/users/sign_in https://github.com/neonbjb/tortoise-tts https://coqui.ai/blog/tts/open_xtts it was even possible 5 years ago https://github.com/CorentinJ/Real-Time-Voice-Cloning

Nope. If you actually tried those, you would quickly find out they don't work. It's actually really hard to clone a voice from a few seconds sample.

A few seconds, yeah. I've seen fairly convincing reproductions from 30 seconds of reading text though.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#138

Earlier quoted context omitted.

Definitely less harmful if only the elites have it. It doesn't make them harder to kill, but it reduces proliferation. And "elites" per se isn't correlated with any political side, so it doesn't even change the culture war. Generally speaking if you think most of the danger of any technology comes from accidents, you want fewer people to have it and don't care too much which ones it is, though it'd be preferable if t…

> And "elites" per se isn't correlated with any political side, so it doesn't even change the culture war. Even if you trust your own government, which I would suggest is misguided (the known propaganda programs that have come out of the DOD alone should be enough to dispel this trust), in a world where only the regulating authorities have access to GenAI, the GRU and the PLA are going to have access to it too. This…

It's not like me having AI is gonna keep the GRU from using it. I'd rather only the GRU have AI; it reduces risk.

The point is not that only the good people should have it, the point is that who has it is less important than as few people have it as possible.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#139
post #90

Earlier quoted context omitted.

I think the other comments make a good argument about how other forms of technology have also degraded trust, but that we've found a way through. I'll also add that I think one potential way we could reinstate trust is through signed multimedia. Cameras/microphones/etc could sign the videos/audio they create in a way that can be used to verify that the media hasn't been doctored. Not sure if that's actually a feasibl…

It's feasible with advanced enough tech. The hard part isn't getting cameras to sign the files they produce. The hard part is to preserve the chain of custody as images are cropped, rescaled, recompressed etc. You can do it with tech like Intel SGX. But you also need serious defense of the camera platforms against hacking, of the CPUs, of the software stacks. And there's no demand. News orgs feel they should be impli…

They might use complicated tech because things are changing due to AI in a way that could degrade their brands trust. By having camera-signed videos, when folks create eg deep fakes of their news anchors/brands, there's a way for consumers to verify what's real. It lets their brands become more trustworthy.

Yeah, preserving the chain of custody is hard. I was thinking there are a few options: (1) the signature of the original video could be attached even after editing/compression, and then a news org would let you look up the signature. That way if someone copied the signature and stuck it on a fake video, you could see the original video that actually passes with that signature, and determine if something has been doctored. Or (2), you could have editing software add a signature verifying the edits made: eg compression, rescaling etc.

And then a law to make it illegal to remove/tamper/valsify a signature, like we have for DVDs, to allow some form of prosecution. The hardware stack is a little easier to protect; the software stack less so. But if we can do it with things like eg browser DRM or http signatures, maybe we can with media editing software? But I'm not versed enough in Cryptography to really know.

And cool will read up on intel SGX, thanks!

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#140

Earlier quoted context omitted.

You and I are just autocomplete machines.

I’m surprised ur the first comment I e seen where a human admits humans aren’t that complex in the big scheme of things

Nothing is complex or important in the big scheme of things, for example when setting the point of reference being the Universe.

But on Earth, humans are pretty complex living organisms that we tag with the term intelligence. And I would say this is the point of reference for the discussion about AI.

I do agree that we are not special and in an alternate Universe maybe a branch of Neanderthals would try to invent some new form of tech.

Post reply on HN