Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NaturalSpeech 2: Zero-shot speech and singing synthesizers

speechresearch.github.io

Post reply on HN
Print view View original

41–50 of 126 posts

Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers

#41

Post by prox »

famouswaffles wrote: post #33
retrac wrote: post #30

This shouldn't be too surprising. Similar results were had with GPT-3 a while back, which is kind-of-able to produce audio or images, encoded as streams of tokens, when trained on that task, despite not being designed for it. A very interesting property was noted a few years ago by multiple researchers, I'm not sure who discovered it first. Transfer learning is unreasonably effective. If you were training an image ge…

There's something a bit more mindblowing than that. Language models and vision models learn representations so similar that you can connect them with just a linear projection between image embedding and text embedding space(no training of the image encoder or llm required). https://arxiv.org/abs/2209.15162 https://llava-vl.github.io/ LLMs are already being grounded.

Do we know how they represent that knowledge? I always hear it called a black box but that seems a bit strange, since you can dissect any data right?

Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers

#42

Post by lukevp »

MattPalmer1086 wrote: post #36
ericd wrote: post #26

Earlier quoted context omitted.

But those certs can come with reputations attached, and it prevents people from claiming that they're representatives of well known companies.

Well, that would create strong financial incentives to compromise company certificates, which I'm sure would then happen. And you still don't know if your taking to a human or an AI.

Like the strong financial incentives to compromise certificates owned by banking websites? Securing voice communications the same as website communications is an awesome idea, and the fact that it’s possible to steal some piece of data and compromise it shouldn’t prevent us from moving in that direction.

Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers

#43

Post by ImHereToVote »

famouswaffles wrote: post #25
ImHereToVote wrote: post #24

Earlier quoted context omitted.

But can it understand why we cry?

Not sure how serious this question is but GPT-4's emotional intelligence is really high.

https://www.youtube.com/watch?v=RKLzSIkCpkw&t=74s

Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers

#44

Post by xutan »

msoad wrote: post #7

Compared to the first NaturalSpeech[1] I'm hearing a lot of white noise in the background. Singing is pretty cool but it feels like we need a few iterations before it can match the ground truth in the way speech does. [1] https://speechresearch.github.io/naturalspeech/

Thanks for your interests in NaturalSpeech and NaturalSpeech 2!

NaturalSpeech focuses on synthesizing human-level high-quality speech, by training on a single-speaker recording-studio dataset.

NaturalSpeech 2 trains on 44K hours of multi-speaker in-the-wild datasets with more than 5K speakers and focuses on synthesizing any speaker's voice in a zero-shot way given only a short speech prompt. When the speech prompt is noisy in the background, NaturalSpeech 2 will mimic this noise as well. If you want clean voice, just give a clean speech prompt is OK.

Check more discussions on reddit as well: https://www.reddit.com/r/singularity/comments/12rubq4/latent...

Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers

#45

Post by MattPalmer1086 »

lukevp wrote: post #42
MattPalmer1086 wrote: post #36

Earlier quoted context omitted.

Well, that would create strong financial incentives to compromise company certificates, which I'm sure would then happen. And you still don't know if your taking to a human or an AI.

Like the strong financial incentives to compromise certificates owned by banking websites? Securing voice communications the same as website communications is an awesome idea, and the fact that it’s possible to steal some piece of data and compromise it shouldn’t prevent us from moving in that direction.

It has happened, but there has to be a big pay off to make it worthwhile.

Smaller companies would likely be more frequent targets given they would be easier to compromise.

By the way, I'm not suggesting that the idea is useless. I'm just pointing out it isn't a panacea, and it still doesn't address the core problem raised in the article that you don't know if you are speaking to a human or not.

Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers

#46

Post by [deleted] »

prox wrote: post #41
famouswaffles wrote: post #33

Earlier quoted context omitted.

There's something a bit more mindblowing than that. Language models and vision models learn representations so similar that you can connect them with just a linear projection between image embedding and text embedding space(no training of the image encoder or llm required). https://arxiv.org/abs/2209.15162 https://llava-vl.github.io/ LLMs are already being grounded.

Do we know how they represent that knowledge? I always hear it called a black box but that seems a bit strange, since you can dissect any data right?

[deleted]

Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers

#47

Post by vimax »

MattPalmer1086 wrote: post #18
XorNot wrote: post #10

Earlier quoted context omitted.

We've had the solution in the form of basic TLS cryptography and verification for decades now though, the problem is no one's implementing it. Governments already maintain registers of legally operating businesses: there's no reason that registration should not also be issuing cryptographic certificates which verify all forms of outbound communication by that business including phone calls. But despite telecom being…

The problem here is authenticating that you are talking to a human and not an AI. Certificates don't help with that. Criminals will register temporary businesses and obtain certificates for them with no problem.

Why does it matter if you’re talking to an AI or a human? What can a criminal AI say that a criminal human cannot?

Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers

#48

Post by MattPalmer1086 »

vimax wrote: post #47
MattPalmer1086 wrote: post #18

Earlier quoted context omitted.

The problem here is authenticating that you are talking to a human and not an AI. Certificates don't help with that. Criminals will register temporary businesses and obtain certificates for them with no problem.

Why does it matter if you’re talking to an AI or a human? What can a criminal AI say that a criminal human cannot?

You can automate it and reach many more victims than doing it with humans.

Edit: it reduces the cost of the attack and thus makes it more profitable.

Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers

#49

Post by hbn »

Speaking of music and AI models - something I thought of yesterday which is an application of AI that would be insanely useful to me is giving it an audio file of a song and having it spit out the chords. I've seen software that attempts to do this in the past but it's all been unimpressive and inaccurate in my testing.

I'm still kinda ignorant with how these models work under the hood, and perhaps that would involve a bunch of new training on music that hasn't been done (and maybe that could be a difficult dataset to train on in terms of copyright). But I play piano, and I can play a song if given the chords, but I'm terrible at transcribing stuff myself. So I'd pay money for a service that does this.

Post reply on HN