Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an end-to-end Speech Recognition model in PyTorch

assemblyai.com

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Building an end-to-end Speech Recognition model in PyTorch

#11

Post by albertzeyer »

option wrote: post #8
albertzeyer wrote: post #4

This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better alternatives. There is also not really a problem about available open source code. There are countless of open source projects which already have that mostly ready to use, for all the common DL frameworks, like TF, PyTorch, Jax, MXNet, whatever. For anyone with…

I question that you need full attention in the acoustic model. The pronunciation of a word in the middle of phrase does not have much dependence on the beginning. You do need attention in the language model part of the pipeline

Almost all the literature which compares CTC and encoder-decoder-attention models shows pretty well that encoder-decoder-attention performs better than CTC in the acoustic model.

See for example here as an overview (my own work, already a bit outdated, but attention has even improved much more since then): https://openreview.net/pdf?id=S1gp9v_jsm

Re: Building an end-to-end Speech Recognition model in PyTorch

#12

Post by coder543 »

Mentioned once in the other comments here without any link, but another open source speech recognition model I heard about recently is Mozilla DeepSpeech:

https://github.com/mozilla/DeepSpeech

https://hacks.mozilla.org/2019/12/deepspeech-0-6-mozillas-sp...

I haven't had a chance to test it, and I wish there were a client-side WASM demo of it that I could just visit on Mozilla's site.

Re: Building an end-to-end Speech Recognition model in PyTorch

#13

Post by tasubotadas »

komuher wrote: post #7

Dunno why (probably dataset) but open source Speech Recognition models are performing very poorly on real world data compared to google speech to text or azure cognitive.

LibreSpeech is 1000h, WSJ is 73h.

Google is training on datasets that are as big as 30kh and MS seems to work on a 10k h dataset.

At the moment, I am working on a similar e2e system but 80h big dataset makes it a really challenging task to generalize well.

Re: Building an end-to-end Speech Recognition model in PyTorch

#14

Post by mikaelphi »

spzb wrote: post #3

This is probably really good but the linked Colab notebook is failing on the first step with some unresolvable dependencies. This does seem to be a bit of a common theme whenever I try running example ML projects. Edit: I think I've fixed it by changing the pip command to: !pip install torchaudio comet_ml==3.0

Author here, thanks for pointing that out is and it's fixed now! I also made sure to add the pip version for torchaudio and torch as well so this should not be an issue anymore.

Re: Building an end-to-end Speech Recognition model in PyTorch

#15

Post by dylanbfox »

albertzeyer wrote: post #4

This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better alternatives. There is also not really a problem about available open source code. There are countless of open source projects which already have that mostly ready to use, for all the common DL frameworks, like TF, PyTorch, Jax, MXNet, whatever. For anyone with…

That's right - most literature does show that encoder-decoder architectures outperform CTC. I think one of the main reasons for this is that CTC assumes the label outputs are conditionally independent of each other, which is a pretty big flaw in that loss function.

The blog does mention Listen-Attend-Spell (which is an encoder-decoder architecture) as an alternative to the CTC model.

Re: Building an end-to-end Speech Recognition model in PyTorch

#16

Post by tasubotadas »

albertzeyer wrote: post #4

This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better alternatives. There is also not really a problem about available open source code. There are countless of open source projects which already have that mostly ready to use, for all the common DL frameworks, like TF, PyTorch, Jax, MXNet, whatever. For anyone with…

It would seem that the best practical approach is to use RNNT as it still lets you do streaming predictions (while Attention won't really let that).

Re: Building an end-to-end Speech Recognition model in PyTorch

#17

Post by mikaelphi »

coder543 wrote: post #12

Mentioned once in the other comments here without any link, but another open source speech recognition model I heard about recently is Mozilla DeepSpeech: https://github.com/mozilla/DeepSpeech https://hacks.mozilla.org/2019/12/deepspeech-0-6-mozillas-sp... I haven't had a chance to test it, and I wish there were a client-side WASM demo of it that I could just visit on Mozilla's site.

Author here! Deep Speech is an excellent repo if you just want to pip install something. We wanted to do a comprehensive writeup to give devs the ability to build their own end-to-end model.

Re: Building an end-to-end Speech Recognition model in PyTorch

#19

Post by bginsburg »

albertzeyer wrote: post #4

This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better alternatives. There is also not really a problem about available open source code. There are countless of open source projects which already have that mostly ready to use, for all the common DL frameworks, like TF, PyTorch, Jax, MXNet, whatever. For anyone with…

Actually end-2-end CTC models are very good - Wav2letter, Jasper, QuartzNet - all these models are much better than DeepSpeech2.

Re: Building an end-to-end Speech Recognition model in PyTorch

#20

Post by bginsburg »

dylanbfox wrote: post #10
komuher wrote: post #7

Dunno why (probably dataset) but open source Speech Recognition models are performing very poorly on real world data compared to google speech to text or azure cognitive.

One of the main factors for this is probably due to dataset size. Commercial STT models are trained on 10s of thousands of hours of data from real-world data. Even a decent model architecture is going to perform pretty well on that much data. Most open source models are trained on Libri, SWB, etc. which are not really big or diverse enough for real-world scenarios. But to max-out results the devil is in the details I…

There are new, very large public English speech datasets: Mozilla Common Voice, National Speech Corpus, which can be combined with LibriSpeech to train large models.
Post reply on HN