Building an end-to-end Speech Recognition model in PyTorch

Good article. Speech recognition for real time use cases must get a really working open source solution. I have been evaluating deepspeech, which is okay. but there is lots of work needed to make it working close to Google Speech engine. Apart from a good Deep neural network, a good speech recognition system needs two important things:

1. Tons of diverse data sets (real world)

2. Solution for Noise - Either de-noise and train OR train with noise.

There are lots of extra challenges that voice recognition problem have to solve which is not common with other deep learning problems:

1. Pitch

2. Speed of conversation

3. Accents (can be solved with more data, I think)

4. Real time inference (low latency)

5. On the edge (i.e. Offline on mobile devices)

This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better alternatives.

There is also not really a problem about available open source code. There are countless of open source projects which already have that mostly ready to use, for all the common DL frameworks, like TF, PyTorch, Jax, MXNet, whatever. For anyone with a bit of ML experience, this should really not be too hard to setup.

But then to get good performance, on your own dataset, what you really need is experience. Probably taking some existing pipeline will get you some model, with an okish word-error-rate. But then you should tune it. In any case, even without tuning, probably encoder-decoder-attention models will perform better than CTC models.

Hah, classic. But in all seriousness I think its a pretty interesting issue. A lot of the ML and data science sees people coming in who do not have formal computer science and software development backgrounds. We build tools and methodologies around abstracting away some of the code development process and hope that it lands us in an environment that's easy to share with others. This is unfortunately rarely the case.

Its a problem that as an industry I think we are in the middle of "solving" (probably can't be solved fully, but things are getting better). I'm really excited to see what kinds of tools and tests will be developed around getting ML projects with some better practices.

I question that you need full attention in the acoustic model. The pronunciation of a word in the middle of phrase does not have much dependence on the beginning.

You do need attention in the language model part of the pipeline

One of the main factors for this is probably due to dataset size. Commercial STT models are trained on 10s of thousands of hours of data from real-world data. Even a decent model architecture is going to perform pretty well on that much data.

Most open source models are trained on Libri, SWB, etc. which are not really big or diverse enough for real-world scenarios.

But to max-out results the devil is in the details IMO (network architecture, optimizer, weight initialization, regularization, data augmentation, hyperparam tuning, etc) which requires a lot of experiments.