Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an end-to-end Speech Recognition model in PyTorch

assemblyai.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Building an end-to-end Speech Recognition model in PyTorch

#21

Post by mikaelphi »

tasubotadas wrote: post #16
albertzeyer wrote: post #4

This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better alternatives. There is also not really a problem about available open source code. There are countless of open source projects which already have that mostly ready to use, for all the common DL frameworks, like TF, PyTorch, Jax, MXNet, whatever. For anyone with…

It would seem that the best practical approach is to use RNNT as it still lets you do streaming predictions (while Attention won't really let that).

Author here! RNNT's is indeed a good approach. RNNT's with masked attention are also capable of streaming as well.

Re: Building an end-to-end Speech Recognition model in PyTorch

#22

Post by ALittleLight »

zerop wrote: post #2

Good article. Speech recognition for real time use cases must get a really working open source solution. I have been evaluating deepspeech, which is okay. but there is lots of work needed to make it working close to Google Speech engine. Apart from a good Deep neural network, a good speech recognition system needs two important things: 1. Tons of diverse data sets (real world) 2. Solution for Noise - Either de-noise…

Your point about needing a dataset made me think about how a post on hackernews like this may be a good way to get data. How many people would contribute by reading a prompt if they visited a link like this and had the option to donate some data? That would get many distinct voices and microphones and some different conditions.

The article mentions that they used a dataset composed of 100 hours of audiobooks. A comment thread here [1] estimates 10-50k visitors from a successful hackernews post. Call it 30k visitors. If 20% of visitors donated by reading a one minute prompt, that's another 6,000 minutes, or, oddly, also 100 hours.

Seems like a potentially easy way to double your dataset and make it more diverse.

1 - https://news.ycombinator.com/item?id=20612717

Re: Building an end-to-end Speech Recognition model in PyTorch

#23

Post by mikaelphi »

albertzeyer wrote: post #4

This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better alternatives. There is also not really a problem about available open source code. There are countless of open source projects which already have that mostly ready to use, for all the common DL frameworks, like TF, PyTorch, Jax, MXNet, whatever. For anyone with…

Author here! CTC models perform quite well and are easy to get started for beginners with an added benefit of real-time streaming capabilities. RNNT's and Encoder-Decoder like Listen-attend-spell are also very solid choices, and literature points to slightly higher accuracy with them on academic datasets. RNNT's being and extension of CTC are streamable as well.

Re: Building an end-to-end Speech Recognition model in PyTorch

#24

Post by Isn0gud »

ALittleLight wrote: post #22
zerop wrote: post #2

Good article. Speech recognition for real time use cases must get a really working open source solution. I have been evaluating deepspeech, which is okay. but there is lots of work needed to make it working close to Google Speech engine. Apart from a good Deep neural network, a good speech recognition system needs two important things: 1. Tons of diverse data sets (real world) 2. Solution for Noise - Either de-noise…

Your point about needing a dataset made me think about how a post on hackernews like this may be a good way to get data. How many people would contribute by reading a prompt if they visited a link like this and had the option to donate some data? That would get many distinct voices and microphones and some different conditions. The article mentions that they used a dataset composed of 100 hours of audiobooks. A comme…

You might be interested in a project, doing exactly that: https://voice.mozilla.org/

Audio data of people reading prompts is quite common, what is missing for robust voice recognition is plenty of data of e.g. people screaming it across the room. There is only so much physics simulations can do.

Re: Building an end-to-end Speech Recognition model in PyTorch

#25

Post by eindiran »

bginsburg wrote: post #20
dylanbfox wrote: post #10

Earlier quoted context omitted.

One of the main factors for this is probably due to dataset size. Commercial STT models are trained on 10s of thousands of hours of data from real-world data. Even a decent model architecture is going to perform pretty well on that much data. Most open source models are trained on Libri, SWB, etc. which are not really big or diverse enough for real-world scenarios. But to max-out results the devil is in the details I…

There are new, very large public English speech datasets: Mozilla Common Voice, National Speech Corpus, which can be combined with LibriSpeech to train large models.

If you combine them you get 5ish K hours of speech for English, which is still fairly small compared to what most big players have access to.

Re: Building an end-to-end Speech Recognition model in PyTorch

#26

Post by albertzeyer »

tasubotadas wrote: post #16
albertzeyer wrote: post #4

This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better alternatives. There is also not really a problem about available open source code. There are countless of open source projects which already have that mostly ready to use, for all the common DL frameworks, like TF, PyTorch, Jax, MXNet, whatever. For anyone with…

It would seem that the best practical approach is to use RNNT as it still lets you do streaming predictions (while Attention won't really let that).

If you need streaming, then yes, RNNT is a good option. If not, encoder-decoder-attention performs a bit better than RNN-T.

Note that there are also approaches for encoder-decoder-attention to make that streaming capable, e.g. MoChA or hard attention, etc.

Google uses RNN-T on-device. But they are researching on extending it with another encoder-decoder-attention model on-top, to get better performance.

This is a quite active research area, and it has not really settled. But CTC is not really too much relevant anymore, as RNNT is just a better variant.

Re: Building an end-to-end Speech Recognition model in PyTorch

#27

Post by solidasparagus »

bginsburg wrote: post #20
dylanbfox wrote: post #10

Earlier quoted context omitted.

One of the main factors for this is probably due to dataset size. Commercial STT models are trained on 10s of thousands of hours of data from real-world data. Even a decent model architecture is going to perform pretty well on that much data. Most open source models are trained on Libri, SWB, etc. which are not really big or diverse enough for real-world scenarios. But to max-out results the devil is in the details I…

There are new, very large public English speech datasets: Mozilla Common Voice, National Speech Corpus, which can be combined with LibriSpeech to train large models.

Amazon worked with 7k hours of labeled data + 1 million hours of unlabeled data - https://arxiv.org/pdf/1904.01624.pdf

Re: Building an end-to-end Speech Recognition model in PyTorch

#28

Post by woodson »

albertzeyer wrote: post #26
tasubotadas wrote: post #16

Earlier quoted context omitted.

It would seem that the best practical approach is to use RNNT as it still lets you do streaming predictions (while Attention won't really let that).

If you need streaming, then yes, RNNT is a good option. If not, encoder-decoder-attention performs a bit better than RNN-T. Note that there are also approaches for encoder-decoder-attention to make that streaming capable, e.g. MoChA or hard attention, etc. Google uses RNN-T on-device. But they are researching on extending it with another encoder-decoder-attention model on-top, to get better performance. This is a qui…

Recent work on transformer transducers with limited right (and left) context seem to give decent results as well: https://arxiv.org/abs/2002.02562

Re: Building an end-to-end Speech Recognition model in PyTorch

#29

Post by woodson »

albertzeyer wrote: post #4

This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better alternatives. There is also not really a problem about available open source code. There are countless of open source projects which already have that mostly ready to use, for all the common DL frameworks, like TF, PyTorch, Jax, MXNet, whatever. For anyone with…

Agreed, and that's where it seems having lots of experience working with speech data helps more than trying to brute-force it with just larger CTC models and "more" data of dubious quality.

Re: Building an end-to-end Speech Recognition model in PyTorch

#30

Post by p1esk »

woodson wrote: post #28
albertzeyer wrote: post #26

Earlier quoted context omitted.

If you need streaming, then yes, RNNT is a good option. If not, encoder-decoder-attention performs a bit better than RNN-T. Note that there are also approaches for encoder-decoder-attention to make that streaming capable, e.g. MoChA or hard attention, etc. Google uses RNN-T on-device. But they are researching on extending it with another encoder-decoder-attention model on-top, to get better performance. This is a qui…

Recent work on transformer transducers with limited right (and left) context seem to give decent results as well: https://arxiv.org/abs/2002.02562

I opened the pdf, did ctrl+F for 'github', got zero results. Have you reproduced their "decent results"?
Post reply on HN