Wav2vec Overview: Semi and Unsupervised Speech Recognition
vaclavkosar.com
Wav2vec Overview: Semi and Unsupervised Speech Recognition
1–10 of 25 posts
Re: Wav2vec Overview: Semi and Unsupervised Speech Recognition
#2One addendum to the linked post's notes:
> SoTa in low-resource setting Libri-light by a lot on WER clean test 100h labeled: others ~4 vs theirs ~2.5
> SoTa on high-resource noisy data (3.3 vs 3.4) close to SoTa on clean data
This note isn't super specific, but it's outdated if I'm understanding it correctly. To my understanding, the SOTA on this data is held by Conformer 1B (a 1 billion parameter model), at 1.4 clean, 2.6 noisy.
Conformer 1B is something like wav2vec 2.0 pretraining + conformer + noisy student + specaugment.
https://arxiv.org/pdf/2010.10504.pdf
--
Wav2vec 2.0 is very cool, but I've had some trouble reproducing the pretraining and fine tuning reliably. It might need a lot of resources (e.g. hundreds of clustered GPUs).
I think Wav2vec-U is extremely cool.
Re: Wav2vec Overview: Semi and Unsupervised Speech Recognition
#3As someone who's an idiot about machine learning, is it possible to run this code in reverse? e.g. take the generated (or novel) vectors and convert them back into audio/waveforms?
Re: Wav2vec Overview: Semi and Unsupervised Speech Recognition
#4As someone who's an idiot about machine learning, is it possible to run this code in reverse? e.g. take the generated (or novel) vectors and convert them back into audio/waveforms?
Generalized reverse projection through even non recurrent neural networks is still an open research problem.
So no in this case.
Re: Wav2vec Overview: Semi and Unsupervised Speech Recognition
#5As someone who's an idiot about machine learning, is it possible to run this code in reverse? e.g. take the generated (or novel) vectors and convert them back into audio/waveforms?
Generalized reverse projection through even non recurrent neural networks is still an open research problem. So no in this case.
That doesn't sound like a particularly realistic problem to solve.
Re: Wav2vec Overview: Semi and Unsupervised Speech Recognition
#6As someone who's an idiot about machine learning, is it possible to run this code in reverse? e.g. take the generated (or novel) vectors and convert them back into audio/waveforms?
[deleted]
Re: Wav2vec Overview: Semi and Unsupervised Speech Recognition
#7As someone who's an idiot about machine learning, is it possible to run this code in reverse? e.g. take the generated (or novel) vectors and convert them back into audio/waveforms?
If you look at the architecture diagram for Wav2Vec-U, the "generator" is doing exactly that - generating waveforms from the vectors. All GANs work this way, and is how websites like https://thispersondoesnotexist.com/ work. Of course as the sibling comment notes the results today might not be great for this task, and it is open research, bit it's not as of it just can't be done at all.
Re: Wav2vec Overview: Semi and Unsupervised Speech Recognition
#8Earlier quoted context omitted.
Generalized reverse projection through even non recurrent neural networks is still an open research problem. So no in this case.
That doesn't sound like a particularly realistic problem to solve.
I agree, but all the more glory if someone does solve it then. And the field is still new enough that I don't want to be cited for decades like the iPod release "no wireless. Less space than a Nomad. Lame." slashdot comment.
Re: Wav2vec Overview: Semi and Unsupervised Speech Recognition
#9As someone who's an idiot about machine learning, is it possible to run this code in reverse? e.g. take the generated (or novel) vectors and convert them back into audio/waveforms?
If you look at the architecture diagram for Wav2Vec-U, the "generator" is doing exactly that - generating waveforms from the vectors. All GANs work this way, and is how websites like https://thispersondoesnotexist.com/ work. Of course as the sibling comment notes the results today might not be great for this task, and it is open research, bit it's not as of it just can't be done at all.
My reading of the generator diagram (figure 6) isn't that it is generating waveforms, but that it is generating phoneme probabilities.
You can train a similar system to produce audio on the output of wav2vec, though it probably won't sound similar to the input audio (accent/voice) unless you expose more features of the input than phonemes.
Re: Wav2vec Overview: Semi and Unsupervised Speech Recognition
#10I wonder how much better this would be at capturing information that doesn't translate well into text representations of speech.
Consider how with word2vec there are relationships in the embedding space between semantically related words. I would expect the examples of that for word2vec (e.g. king -> queen being a similar translation as man -> woman) to apply here too, but can it also do things like place regular questions and rhetorical questions in different regions of the embedding space based off of of the inflection in the speech?
It would also be interesting to see what relationships exist between equivalent words in different languages within the embedding space. I suppose something like that is probably already used for text translation neural networks, but maybe some notable differences exist when dealing with speech directly.