Oh I'm super excited about singing synths! I write mainly for choirs, and it'll be a game changer to have software actually perform this music!
NaturalSpeech 2: Zero-shot speech and singing synthesizers
51–60 of 126 posts
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#52Speaking of music and AI models - something I thought of yesterday which is an application of AI that would be insanely useful to me is giving it an audio file of a song and having it spit out the chords. I've seen software that attempts to do this in the past but it's all been unimpressive and inaccurate in my testing. I'm still kinda ignorant with how these models work under the hood, and perhaps that would involve…
This actually exists! Check out fadr.com. They isolate the MIDI, drums, bass, vocals, and more for you.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#53Earlier quoted context omitted.
There's something a bit more mindblowing than that. Language models and vision models learn representations so similar that you can connect them with just a linear projection between image embedding and text embedding space(no training of the image encoder or llm required). https://arxiv.org/abs/2209.15162 https://llava-vl.github.io/ LLMs are already being grounded.
Do we know how they represent that knowledge? I always hear it called a black box but that seems a bit strange, since you can dissect any data right?
No we don't know how they represent that knowledge. But performing experiments to probe how similar they are is a lot easier than knowing all that.
They're called black boxes because we can't explain what the weights are learning during training, what the different weights do or are responsible for to shift or produce the output it does.
It's like, biologists know how neurons communicate signals with each other. But is that knowledge enough to explain human behavior ? Not even close.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#54Earlier quoted context omitted.
The problem here is authenticating that you are talking to a human and not an AI. Certificates don't help with that. Criminals will register temporary businesses and obtain certificates for them with no problem.
Why does it matter if you’re talking to an AI or a human? What can a criminal AI say that a criminal human cannot?
[dead]
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#55Are NaturalSpeech or NaturalSpeech 2 from the research open source and/or available for playing with? I see one implementation[1] but it seems to be from a third party (that might be totally fine, but wondering if there's an "official").
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#56This shouldn't be too surprising. Similar results were had with GPT-3 a while back, which is kind-of-able to produce audio or images, encoded as streams of tokens, when trained on that task, despite not being designed for it. A very interesting property was noted a few years ago by multiple researchers, I'm not sure who discovered it first. Transfer learning is unreasonably effective. If you were training an image ge…
There's something a bit more mindblowing than that. Language models and vision models learn representations so similar that you can connect them with just a linear projection between image embedding and text embedding space(no training of the image encoder or llm required). https://arxiv.org/abs/2209.15162 https://llava-vl.github.io/ LLMs are already being grounded.
We've known that embeddings have this property since the glove paper at least. Linear substructure in low dimensional representations of ultra high dimensions is shockingly common
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#57Earlier quoted context omitted.
Do we know how they represent that knowledge? I always hear it called a black box but that seems a bit strange, since you can dissect any data right?
No we don't know how they represent that knowledge. But performing experiments to probe how similar they are is a lot easier than knowing all that. They're called black boxes because we can't explain what the weights are learning during training, what the different weights do or are responsible for to shift or produce the output it does. It's like, biologists know how neurons communicate signals with each other. But…
Interesting, so it could be similar to how our brains store knowledge, but could also be completely different.
This makes me wonder if these models are perfect universal translators once they “grasp” a concept.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#58Earlier quoted context omitted.
The problem here is authenticating that you are talking to a human and not an AI. Certificates don't help with that. Criminals will register temporary businesses and obtain certificates for them with no problem.
Why does it matter if you’re talking to an AI or a human? What can a criminal AI say that a criminal human cannot?
A criminal human probably can't do a perfect voice impersonation of your teenage kid or frail grandmother who desperately needs to be sent money for some urgent reason.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#59Is it just me, or the accompanying text does not correspond to the audio?
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#60Does anyone know how many words would correspond to the diffusion model's batch size of 6000 frames?