Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT unexpectedly began speaking in a user's cloned voice during testing

arstechnica.com

Post reply on HN
Print view View original

91–100 of 164 posts

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#91

Post by b112 »

squigz wrote: post #77
c0mbonat0r wrote: post #38

Earlier quoted context omitted.

Can someone pleasee convince why i shouldn't be absolutely shit out of my mind cynical about this innovation? we are literally seeing the downfall of trust in society. and no, i dont believe i am exaggerating

We have adapted to monumental shifts in how we develop trust in society for as long as society has existed - from the printing press to photography to the Internet to CGI to ... I don't see this as any different. We will determine new ways of establishing trust. They'll certainly have flaws, as establishing trust in a society always has, but we'll learn to recognize those flaws and hopefully fix them. Beyond that, wh…

The big difference is, there was a very high bar to forging photographs, and most news was gated (with the ability to easily find, and sue those guilty of slander/libel).

Now it's utterly simplistic to forge, to libel, to slander, and there is no easy path in many cases to sue.

While you can say "yes, but..." to the above, that's the reality that we've lived with for 150 years, less extremely rare edge cases. All this has changed over the course of a couple of decades, with most of that change in the last 10, and focuses on the last 2 years.

Beyond that, it took significant effort and labour to create fake stories and images. People had to be experts, or be wordsmiths. Now, click click, and fake generated stories abound. In fact, they're literally everywhere. There's absolutely no comparison here.

Now, in the time it used to take one person to generate one fake story, you can generate millions and trillions if you have the cash. Really, it's the same problem with spam phone calls, and with spam email.

You didn't get 1000 spam letters in the mail in the 80s, because that cost money. Email was free, thus spam became plentiful. The same with spam phone calls, it cost hard cash for each call, now it's pennies per hundreds of automated calls, so spam phone calls abound.

The same is happening with all content on the internet. Realistically the web is now dead. It's now gone. Even things such as wikipedia are going to die, as over the next 2 to 3 years LLM output will become utterly and completely indistinguishable in all aspects.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#92

Post by stavros »

lolinder wrote: post #84
wyager wrote: post #78

Earlier quoted context omitted.

You are making a very popular mistake: confusing the training procedure for LLMs (autocomplete) with "how they work"/their internal ontology (mostly unknown). When we teach children how to do arithmetic, we have them predict missing items in equations. We don't accuse them of "only doing autocomplete". The same applies for LLMs.

I was implementing my own transformer-based models and fine tuning GPT-2 in 2019, and I've kept up with every development since then. I understand the internal structure of these things better than nearly all of the "AI Engineers" who are currently working on wrapping them up as black boxes embedded in applications. I'm not making a "popular mistake", I'm literally describing how inference is done.

Yes, but it's akin to describing humans as just "interactions between molecules". Technically true, but useless.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#93

Post by AmericanChopper »

tsimionescu wrote: post #79
AmericanChopper wrote: post #60

Earlier quoted context omitted.

The way I see it LLMs are just making it more obvious to see all the flaws with the existing levels of trust. Humans have never had access to universal truths, or universal ways of validating anything. Any claim anybody makes could be intentionally or unintentionally deceitful or untrue. The idea that there are sources you can trust to do your thinking for you is the more dangerous illusion in my opinion, and I’m not…

> The idea that there are sources you can trust to do your thinking for you is the more dangerous illusion in my opinion, and I’m not convinced that society will be harmed by poking some holes through it There is no alternative to this idea. It is completely impossible for an individual to possess all of the knowledge of everything that affects their lives. The only option for getting some of this information is goin…

> There is no alternative to this idea.

I said do your thinking for you, not do your information gathering for you.

I would suggest that you do not trust any single source to only ever tell you things that are true. If there’s a topic you want to know something about it’s a much better course of action to look at multiple different sources, and do your own thinking to come to your own conclusions.

There are no authorities who can reliably take on this role for you, and LLMs don’t change this. The same is true with science. Even prior to LLMs, the replication crisis should have shown that a single paper on any topic can’t be relied on to contain any truth (the same would be true even in the absence of a replication crisis for that matter).

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#94

Post by wyager »

lolinder wrote: post #84
wyager wrote: post #78

Earlier quoted context omitted.

You are making a very popular mistake: confusing the training procedure for LLMs (autocomplete) with "how they work"/their internal ontology (mostly unknown). When we teach children how to do arithmetic, we have them predict missing items in equations. We don't accuse them of "only doing autocomplete". The same applies for LLMs.

I was implementing my own transformer-based models and fine tuning GPT-2 in 2019, and I've kept up with every development since then. I understand the internal structure of these things better than nearly all of the "AI Engineers" who are currently working on wrapping them up as black boxes embedded in applications. I'm not making a "popular mistake", I'm literally describing how inference is done.

> I'm literally describing how inference is done.

You are very clearly not doing that. Nothing about your comment had anything to do with the internal structure of LLMs.

I believe you that you've set up some models with pytorch or whatever, but this seemingly hasn't translated to a sufficiently coherent mental model to make the distinction between the extrinsic optimization criterion and intrinsic behavior.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#95

Post by stefan_ »

sigmoid10 wrote: post #7

This problem appeared during pre-release testing and has since been solved post-generation using an output classifier that verifies responses, according to the system card release. It was predictable that someone would spin this into a black mirror-esque clickbait story.

It would be more convincing if "has since been solved post-generation using an output classifier that verifies responses, according to the system card release" didn't sound like an AutoNation certified pre-owned car; the blind certifying something they neither understand nor control, but it sure checks all the boxes on the "model card".

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#96

Post by danielmarkbruce »

lolinder wrote: post #72
skybrian wrote: post #27

It’s “unexpected” because their early training didn’t get rid of it as well as they hoped. LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. For now, it’s fairly harmless since it’s only a blooper in a lab, but there will likely be open-weights versions of this sort of thing eventually. And there will probably be p…

> LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. This is a great summary of almost everything that goes wrong with LLM applications. LLMs are autocomplete machines, which is why GitHub Copilot is still the most reliably useful application of LLM tech out there. The further you get from autocomplete, the less rel…

You and I are just autocomplete machines.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#97

Post by samtho »

wyager wrote: post #78
lolinder wrote: post #72

Earlier quoted context omitted.

> LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. This is a great summary of almost everything that goes wrong with LLM applications. LLMs are autocomplete machines, which is why GitHub Copilot is still the most reliably useful application of LLM tech out there. The further you get from autocomplete, the less rel…

You are making a very popular mistake: confusing the training procedure for LLMs (autocomplete) with "how they work"/their internal ontology (mostly unknown). When we teach children how to do arithmetic, we have them predict missing items in equations. We don't accuse them of "only doing autocomplete". The same applies for LLMs.

This is only a common mistake to the subset of people who labor under the delusion that we have actually stumbled upon Generalized Artificial Intelligence.

We know how LLMs work fundamentally and what their limits are. LLMs are only able to make “correct sounding” statements which have the side effect of being correct a certain percentage of the time. They do not have the ability to reason nor engage in high level thought.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#98

Post by complaintdept »

ttul wrote: post #80
lolinder wrote: post #72

Earlier quoted context omitted.

> LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. This is a great summary of almost everything that goes wrong with LLM applications. LLMs are autocomplete machines, which is why GitHub Copilot is still the most reliably useful application of LLM tech out there. The further you get from autocomplete, the less rel…

I'd respectfully disagree with this characterization of LLMs. While they certainly excel at pattern recognition, calling them mere "autocomplete machines" vastly undersells their capabilities. LLMs demonstrate complex reasoning, multi-modal understanding, and emergent behaviors that go well beyond simple pattern continuation. They've succeeded in areas like mathematical problem-solving, creative tasks, and various re…

> LLMs demonstrate complex reasoning, multi-modal understanding, and emergent behaviors that go well beyond simple pattern continuation.

Can they win Wordle yet?

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#99

Post by lolinder »

wyager wrote: post #94
lolinder wrote: post #84

Earlier quoted context omitted.

I was implementing my own transformer-based models and fine tuning GPT-2 in 2019, and I've kept up with every development since then. I understand the internal structure of these things better than nearly all of the "AI Engineers" who are currently working on wrapping them up as black boxes embedded in applications. I'm not making a "popular mistake", I'm literally describing how inference is done.

> I'm literally describing how inference is done. You are very clearly not doing that. Nothing about your comment had anything to do with the internal structure of LLMs. I believe you that you've set up some models with pytorch or whatever, but this seemingly hasn't translated to a sufficiently coherent mental model to make the distinction between the extrinsic optimization criterion and intrinsic behavior.

I'm not talking about optimization criteria or training, I'm talking about how we use the model for inference.

We feed in a context and it gives a probability distribution for the next word. We sample from that distribution following some set of rules. We then update the context with the new word and repeat.

That algorithm is an autocomplete algorithm. As long as that's what LLM inference looks like, all problems that we want to feed to an LLM therefore must be translated to autocomplete.

I think you're making the mistake of assuming that because I use language that resembles the language used by cynics I'm therefore arguing that LLMs are useless. I'm not. All I'm saying is that we need to have an accurate mental model for the way these things work, and that mental model is autocomplete. Nearly every major failure in an LLM application was the result of failing to keep that in mind.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#100

Post by billti »

lolinder wrote: post #86
ttul wrote: post #80

Earlier quoted context omitted.

I'd respectfully disagree with this characterization of LLMs. While they certainly excel at pattern recognition, calling them mere "autocomplete machines" vastly undersells their capabilities. LLMs demonstrate complex reasoning, multi-modal understanding, and emergent behaviors that go well beyond simple pattern continuation. They've succeeded in areas like mathematical problem-solving, creative tasks, and various re…

I never said that we couldn't use them for other things—it turns out that a lot of problems can be translated to autocomplete! And yes, in many cases the loss can be made minimal by being clever with the translation. What I'm saying is that nearly every failure of LLM applications boils down to the engineers not understanding that they're just autocomplete engines. Most new LLM products are put together by someone wh…

In that vein, posting comments to hackers news is just an auto-complete, or “what should the next token be” problem, right? :-)

Yet here we all are, reading the comments, looking for insights, knowledge, and entertainment.

Post reply on HN