Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSpeech: Scaling up end-to-end speech recognition

arxiv.org

Post reply on HN
Print view View original

11–20 of 23 posts

Re: DeepSpeech: Scaling up end-to-end speech recognition

#11

Post by pesenti »

cbcase wrote: post #9
pesenti wrote: post #7

To put it in perspective, my team in IBM Watson has already published better numbers (10.4% WER vs 13.1% WER for Baidu) on the SWB dataset. We haven't run our model on the CH part so we can't compare on the full test set. Paper here: http://www.mirlab.org/conference_papers/International_Confer... .

Hi Jerome, those are great results! We got an email this morning from someone else on the Watson team pointing out that we didn't include the latest IBM number -- we'll be sure to update the results in the next version of the paper (three cheers for arXiv). Of course, we openly say in the paper that we don't have the best result on easy subset of Hub5'00 (we had it as 11.5%). We're more interested in advancing the st…

The team is already working on seeing what we get with CH. We'll let you know where we land. But your results are definitely impressive. We love to see new published innovation in the field. Kudos to the team!

Re: DeepSpeech: Scaling up end-to-end speech recognition

#12

Post by greeneggs »

Very nice. I wonder if training can be simplified by training pieces of the model separately, instead of training all together. For example, the DeepSpeech model has three layers of feedforward neurons (where the inputs to the first layer are overlapping contexts of audio), followed by a bi-directional recurrent layer, followed by another feedforward layer. What would the results be if we trained the first layers (perhaps all three) on a different problem, such as autoencoding or fill-in-the-blank (as in word2vec), and then fixed those network weights to train the rest of the network?

Breaking the network up like this would reduce training time and perhaps reduce the needed training data. Since the first layers could be trained without supervision, less labeled data would be needed to train the last two layers. It would also facilitate transferring models between problems; the output of the first few layers, like a word2vec, could be fed into arbitrary other machine learning problems, e.g., translation.

If this does not work, then how about training the whole model together, but only once? The final results are reported for an ensemble of six independently trained networks. What if started by training one network, and then fixed the first three layers to train other networks? (Instead of fixing the first layers, you could also just give them a slower training rate, although it isn't clear whether that would save you much.)

Re: DeepSpeech: Scaling up end-to-end speech recognition

#13

Post by gok »

So with 300 hours of training data it does worse on SWB than a DNN-HMM, or even a GMM-HMM system? But when they give it 2300 hours or training data, it can beat those 300 hour trained systems?

This is still very cool, but that comparison doesn't seem fair at all.

Re: DeepSpeech: Scaling up end-to-end speech recognition

#14

Post by dingdingdang »

gok wrote: post #13

So with 300 hours of training data it does worse on SWB than a DNN-HMM, or even a GMM-HMM system? But when they give it 2300 hours or training data, it can beat those 300 hour trained systems? This is still very cool, but that comparison doesn't seem fair at all.

Are any of these systems open source?

Re: DeepSpeech: Scaling up end-to-end speech recognition

#15

Post by sherjilozair »

gok wrote: post #13

So with 300 hours of training data it does worse on SWB than a DNN-HMM, or even a GMM-HMM system? But when they give it 2300 hours or training data, it can beat those 300 hour trained systems? This is still very cool, but that comparison doesn't seem fair at all.

Why not? DNN-HMM and GMM-HMM won;t have done any better even if trained for 2300 hours.

Re: DeepSpeech: Scaling up end-to-end speech recognition

#16

Post by cbcase »

FractalNerve wrote: post #8
cbcase wrote: post #3

Earlier quoted context omitted.

I should add that I had the opportunity to work on this project and am happy to answer questions.

How much latency does the system have in the best/worst and average case? And is your implementation public?

For a single utterance, it's fast enough that we can produce results in real time. Of course, building a production system for millions of users might require just a bit more engineering work...

Re: DeepSpeech: Scaling up end-to-end speech recognition

#17

Post by cbcase »

sherjilozair wrote: post #15
gok wrote: post #13

So with 300 hours of training data it does worse on SWB than a DNN-HMM, or even a GMM-HMM system? But when they give it 2300 hours or training data, it can beat those 300 hour trained systems? This is still very cool, but that comparison doesn't seem fair at all.

Why not? DNN-HMM and GMM-HMM won;t have done any better even if trained for 2300 hours.

Mostly this, though it's not so black-and-white. The paper discusses results from a DNN-HMM system (Maas et al., using Kaldi) trained on 2k hours, and it does provide a small generalization improvement over 300 hours.

Much of the excitement about deep learning -- which we see as well in DeepSpeech -- is that these models continue to improve as we provide more training data. It's not obvious a priori that results will keep getting better after thousands of hours of speech. We're exited to keep advancing that frontier.

Re: DeepSpeech: Scaling up end-to-end speech recognition

#18

Post by cbcase »

dingdingdang wrote: post #14
gok wrote: post #13

So with 300 hours of training data it does worse on SWB than a DNN-HMM, or even a GMM-HMM system? But when they give it 2300 hours or training data, it can beat those 300 hour trained systems? This is still very cool, but that comparison doesn't seem fair at all.

Are any of these systems open source?

Both Kaldi[1] and CMU Sphinx[2] are high-quality open source speech systems. I know for a fact that Kaldi includes support for DNN acoustic models (I'm less familiar with Sphinx).

[1] http://kaldi.sourceforge.net/ [2] http://cmusphinx.sourceforge.net/

Re: DeepSpeech: Scaling up end-to-end speech recognition

#19

Post by cbcase »

brandonb wrote: post #4

This is very fast progress from Baidu's Silicon Valley AI lab! Andrew Ng only joined Baidu in May, and (nearly?) all of the co-authors of this paper have joined him since then: http://www.technologyreview.com/news/527301/chinese-search-g... Congrats to Carl, Sanjeev, Andrew, and the others.

Thanks for the kind words, Brandon! Been a busy couple of months :)

Re: DeepSpeech: Scaling up end-to-end speech recognition

#20

Post by gok »

cbcase wrote: post #17
sherjilozair wrote: post #15

Earlier quoted context omitted.

Why not? DNN-HMM and GMM-HMM won;t have done any better even if trained for 2300 hours.

Mostly this, though it's not so black-and-white. The paper discusses results from a DNN-HMM system (Maas et al., using Kaldi) trained on 2k hours, and it does provide a small generalization improvement over 300 hours. Much of the excitement about deep learning -- which we see as well in DeepSpeech -- is that these models continue to improve as we provide more training data. It's not obvious a priori that results will…

That was an even weirder comparison. They compare a system trained on 2000 hours of acoustic data mismatched with the testing data to their system, which was trained on 300 hours of matched data in addition to the 2000 hours of mismatched acoustic data.
Post reply on HN