I'm guessing emotional pre-prompts are difficult. The current offerings like ElevenLabs and WellSaidLabs provide amazing voices for narration but lack any way to change the emotions (e.g. happy, angry, excited, etc.). I wonder what are the technical hurdles to adding this variability?
NaturalSpeech 2: Zero-shot speech and singing synthesizers
21–30 of 126 posts
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#22Only a three second sample for in context learning needed to scam people now, very impressive.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#23Is the approach similar to what they did for Riffusion? (On front page a while ago)
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#24Earlier quoted context omitted.
We aren't even far off from an LLM being able to infer that the parents are fake on the basis of the dog name. I'm not even gonna touch the chain gun shooting up a parking lot aspect.
Not Far off ? GPT-4 already gets it. Response after pasting in the scene: The T-800 figured out that John's foster parents were dead based on the exchange because when it asked about "Wolfie" (a made-up name for the dog), the T-1000, impersonating Janelle, did not correct the name and instead went along with it, saying "Wolfie's fine." If the real Janelle had been on the phone, she would have corrected the T-800 by s…
But can it understand why we cry?
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#25Earlier quoted context omitted.
Not Far off ? GPT-4 already gets it. Response after pasting in the scene: The T-800 figured out that John's foster parents were dead based on the exchange because when it asked about "Wolfie" (a made-up name for the dog), the T-1000, impersonating Janelle, did not correct the name and instead went along with it, saying "Wolfie's fine." If the real Janelle had been on the phone, she would have corrected the T-800 by s…
But can it understand why we cry?
Not sure how serious this question is but GPT-4's emotional intelligence is really high.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#26Earlier quoted context omitted.
We've had the solution in the form of basic TLS cryptography and verification for decades now though, the problem is no one's implementing it. Governments already maintain registers of legally operating businesses: there's no reason that registration should not also be issuing cryptographic certificates which verify all forms of outbound communication by that business including phone calls. But despite telecom being…
The problem here is authenticating that you are talking to a human and not an AI. Certificates don't help with that. Criminals will register temporary businesses and obtain certificates for them with no problem.
But those certs can come with reputations attached, and it prevents people from claiming that they're representatives of well known companies.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#27Earlier quoted context omitted.
We've had the solution in the form of basic TLS cryptography and verification for decades now though, the problem is no one's implementing it. Governments already maintain registers of legally operating businesses: there's no reason that registration should not also be issuing cryptographic certificates which verify all forms of outbound communication by that business including phone calls. But despite telecom being…
And then the government will get hacked and the data leaked.
Familiarize yourself with a certificate signing request.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#28Wow, that ethics statement at the end
Yep. Tell me 1986 self that this is real and see what I say:
"We will always take Microsoft AI Principles as guidelines to develop such AI models"
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#29Wow, that ethics statement at the end
Yes, more "please don't be bad" than an ethics statement.
you made me look:
"NaturalSpeech 2 can synthesize speech with good expressiveness/fidelity and good similarity with a speech prompt, which could be potentially misused, such as speaker mimicking and voice spoofing. To avoid potential issues, we appeal to our practitioners to not abuse this technology and to develop defending tools to detect AI-synthesized voices. We will always take Microsoft AI Principles as guidelines to develop such AI models."
Microsoft Responsible AI Standard, v2 (General Requirements 2022)
https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE...
That has a nice section called "Goal A2: Oversight of significant adverse impacts", which says "Microsoft AI systems are reviewed to identify systems that may have a significant adverse impact on people, organizations, and society, and additional oversight and requirements are applied to those systems."
I couldn't find the Impact Assessment for this tech released by Microsoft. How did the 'Natural Speech 2' fare in this review process? Where is the report?
https://www.bing.com/search?q=Responsible+AI+Impact+Assessme...
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#30This shouldn't be too surprising. Similar results were had with GPT-3 a while back, which is kind-of-able to produce audio or images, encoded as streams of tokens, when trained on that task, despite not being designed for it.
A very interesting property was noted a few years ago by multiple researchers, I'm not sure who discovered it first. Transfer learning is unreasonably effective. If you were training an image generator network, there's a significant reduction in training time, by taking a model already trained and fine-tuning it, compared to starting from a model with truly random weights.
This isn't surprising when we're talking photos of ambulances and moving to photos of trucks. But it holds true when you train it on ... well, anything structured, really. A GPT-style transformer trained on online comments, or audio samples of music encoded as token streams, when switched to images of cars encoded as token streams, learns that task much more quickly than if it had been fully randomized.
I don't see how to escape the conclusion that these models learn some sort of general properties (something about arithmetic and mathematical relationships, maybe?) There's some sort of abstraction or internal model that is learned, that is applicable across very different tasks.