So who will be developing an open source assistant like Goggle Home / Alexa? :)
If you do a 1 minute google search you will find that there are multple open source voice assistant offerings already, for example mycroft.
DeepSpeech 0.6
21–30 of 68 posts
Re: DeepSpeech 0.6
#22I do not understand how to use Deepspeech even in the most simple use case. 1. I want to teach it ten words. How do I do this? 2. I want to speak into my microphone (available as a Pulseaudio device) and recognise the words and output the words as a text stream on stdout. How do I do this? This is the documentation: https://deepspeech.readthedocs.io/en/v0.6.0/Python-Examples.... https://deepspeech.readthedocs.io/en/v…
> 1. I want to teach it ten words. How do I do this?
I'm assuming you mean "my target vocabulary is only ten words". In that case, you have a couple of options:
1. Just use it as is and see how it does.
2. Create a language model containing just those 10 words so that the engine is constrained to that vocabulary, which should improve accuracy.
3. Train a (smaller) model from scratch, which requires (audio, transcript) pairs. This is basically research work, there's no recipe you can just follow. The advantage is that if you can make it work, it can be a much smaller model than a model trained for general English dictation.
> 2. I want to speak into my microphone (available as a Pulseaudio device) and recognise the words and output the words as a text stream on stdout. How do I do this?
We have a few examples with microphone input, but they're all based on voice activity detection: https://github.com/mozilla/DeepSpeech/tree/master/examples
I should write an example that is just continuous streaming and output without any voice activity detection.
I'd be happy to discuss further on Discourse: https://discourse.mozilla.org/c/deep-speech
Re: DeepSpeech 0.6
#23> It achieves a 7.5% word error rate on the LibriSpeech test clean benchmark Anyone have a comparison for how good/bad that is compared to other solutions, and what it means for practical usage, if that can be guessed at from a single number?
Author here. I would add to what the sibling comments have mentioned by saying that SotA results should be taken with a grain of salt. Our engine is capable of streaming (processing the audio as it's being recorded), which is not doable with architectures that have bidirectional decoders or attention mechanisms that require the whole encoder input ahead of time. For real world applications, this is absolutely crucial…
Hi! I'm from the RWTH team (https://paperswithcode.com/sota/speech-recognition-on-libris...). Our best system (2.3% WER) (trained only on the Librispeech data, i.e. much less data than your model) currently is a hybrid HMM/NN model, and you are right, the acoustic model uses a BLSTM. However, we have shown in other work (https://www-i6.informatik.rwth-aachen.de/publications/downlo...) that you can expect similar performance in an online system, or not much worse (there is also many related work, which often is maybe 5-10% relative worse than the offline system).
I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right? As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants. They should give you all the same properties (online capable, low latency, simple, end-to-end), but much better WER performance. Or is this simply because there currently is no simple ready-to-use implementation for those? Note that we published some TF code recently for some streaming attention variants, and plan to publish some RNN-T/RNA code soon.
Re: DeepSpeech 0.6
#24Earlier quoted context omitted.
Author here. I would add to what the sibling comments have mentioned by saying that SotA results should be taken with a grain of salt. Our engine is capable of streaming (processing the audio as it's being recorded), which is not doable with architectures that have bidirectional decoders or attention mechanisms that require the whole encoder input ahead of time. For real world applications, this is absolutely crucial…
Hi! I'm from the RWTH team ( https://paperswithcode.com/sota/speech-recognition-on-libris... ). Our best system (2.3% WER) (trained only on the Librispeech data, i.e. much less data than your model) currently is a hybrid HMM/NN model, and you are right, the acoustic model uses a BLSTM. However, we have shown in other work ( https://www-i6.informatik.rwth-aachen.de/publications/downlo... ) that you can expect similar…
Hi!
> I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right?
Simplicity and ease of targeting other languages, yes. We're a small team.
> As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants.
We started DeepSpeech in 2016, before these recent developments for end-to-end ASR were mainstream/SotA.
> Or is this simply because there currently is no simple ready-to-use implementation for those?
Implementing the model architecture for training is only part of the problem for us. We have a hand-crafted inference graph to allow for small and efficient client code and inference models, and the more complex the architecture is, the trickier it gets to make sure it all works on all platforms, including TFLite, with quantization, etc.
We're investigating alternative architectures as well as mixed CTC/RNN LMs to deal with language model size, but no final decisions made yet.
> Note that we published some TF code recently for some streaming attention variants, and plan to publish some RNN-T/RNA code soon.
Nice! Can you share a link to the streaming attention code?
Re: DeepSpeech 0.6
#25I do not understand how to use Deepspeech even in the most simple use case. 1. I want to teach it ten words. How do I do this? 2. I want to speak into my microphone (available as a Pulseaudio device) and recognise the words and output the words as a text stream on stdout. How do I do this? This is the documentation: https://deepspeech.readthedocs.io/en/v0.6.0/Python-Examples.... https://deepspeech.readthedocs.io/en/v…
> 1. I want to teach it ten words. How do I do this? I'm assuming you mean "my target vocabulary is only ten words". In that case, you have a couple of options: 1. Just use it as is and see how it does. 2. Create a language model containing just those 10 words so that the engine is constrained to that vocabulary, which should improve accuracy. 3. Train a (smaller) model from scratch, which requires (audio, transcript…
>Just use it as is and see how it does.
How? The docs seem to be suffering from the same problem as the git docs: they assume the reader is already a domain expert, which would seem to run counter to the stated goal of being simple (and ubiquitous).
I tried to figure out how to "just use" deepspeech and failed.
Re: DeepSpeech 0.6
#26I just found https://voice.mozilla.org/ Besides being a great resource for speech analysis, this could be a real game changer for acquiring listening comprehension in a foreign language. I feel that even after a few years of learning a new language I still have trouble with listening. Part of that is that it's often all or nothing, even one or two unknown words in a sentence means I can't understand the sentence. But…
Some of the English samples by non-natives were completely incomprehensible. I hope the system learns to be accent free.
Re: DeepSpeech 0.6
#27I just found https://voice.mozilla.org/ Besides being a great resource for speech analysis, this could be a real game changer for acquiring listening comprehension in a foreign language. I feel that even after a few years of learning a new language I still have trouble with listening. Part of that is that it's often all or nothing, even one or two unknown words in a sentence means I can't understand the sentence. But…
Some of the English samples by non-natives were completely incomprehensible. I hope the system learns to be accent free.
They have a validation process, and the data is mainly meant to train speech-to-text.
For that, it's important to include accents and even non-native speakers, especially in English.
I think that those wavenet models for TTS were able to be conditioned by gender and maybe accent, if they had the data.
Re: DeepSpeech 0.6
#28Earlier quoted context omitted.
> 1. I want to teach it ten words. How do I do this? I'm assuming you mean "my target vocabulary is only ten words". In that case, you have a couple of options: 1. Just use it as is and see how it does. 2. Create a language model containing just those 10 words so that the engine is constrained to that vocabulary, which should improve accuracy. 3. Train a (smaller) model from scratch, which requires (audio, transcript…
>Just use it as is and see how it does. How? The docs seem to be suffering from the same problem as the git docs: they assume the reader is already a domain expert, which would seem to run counter to the stated goal of being simple (and ubiquitous). I tried to figure out how to "just use" deepspeech and failed.
There's a basic example here transcribing from an audio file: https://github.com/mozilla/DeepSpeech/tree/v0.6.0/#project-d...
There are also examples transcribing from the microphone as I mentioned above: https://github.com/mozilla/DeepSpeech/tree/master/examples/
I also have a small GUI example here [0], click once to start recording, once again to stop and show transcript. It receives the same command line arguments as the ones shown in the README link above (namely --model, --lm and --trie).
The README starts with a very basic installation and usage guide, I don't see how that's assuming any expertise.
Re: DeepSpeech 0.6
#29Earlier quoted context omitted.
Hi! I'm from the RWTH team ( https://paperswithcode.com/sota/speech-recognition-on-libris... ). Our best system (2.3% WER) (trained only on the Librispeech data, i.e. much less data than your model) currently is a hybrid HMM/NN model, and you are right, the acoustic model uses a BLSTM. However, we have shown in other work ( https://www-i6.informatik.rwth-aachen.de/publications/downlo... ) that you can expect similar…
Hi! > I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right? Simplicity and ease of targeting other languages, yes. We're a small team. > As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants. We started DeepSpeech in 2016, before these recent developments for end…
I really hope you adopt the latest models particularly streaming attention variants. I think you should validate with users the assumption that latency is more important than WER.
IMHO the WER is more important than latency improvements in the millisecond range. The most frustrating thing is having to dictate over and over and the transcription is incorrect each time.
Consider that the time to a correct transcription is the latency plus error correction. If error correction is manual it will be orders of magnitude slower, so optimize for WER.
I’m terms of competition, Siri has latency in the 5+ second range due to the network call especially in area with poor data rates. I think a client side model like yours will easily win in this category. If you’re already ahead here, why not focus on WER next?
Another great capability is to generate alternative transcriptions for words with low confidence values to allow for quick error correction. Do you offer something like this today?
Also, consider the long term view that new models are constantly being released and refined. It’d be best to have an architecture that allows quick replacement without a lot of hand tuning, or where the tuning can be automated to a greater extent.
Re: DeepSpeech 0.6
#30I do not understand how to use Deepspeech even in the most simple use case. 1. I want to teach it ten words. How do I do this? 2. I want to speak into my microphone (available as a Pulseaudio device) and recognise the words and output the words as a text stream on stdout. How do I do this? This is the documentation: https://deepspeech.readthedocs.io/en/v0.6.0/Python-Examples.... https://deepspeech.readthedocs.io/en/v…
I think you're being a little harsh.
1. Train it using the training dataset but change all word labels that aren't in your 10 word list to "OTHER_WORD" or whatever. There's probably not much point doing this. Alternatively I guess you can restrict the beam search to only look for your 10 words. Yes this is really complicated - it's a speech recognition engine - they are really complicated!
2. The API docs look pretty clear to me:
> createStream()[source]
> Create a new streaming inference state. The streaming state returned by this function can then be passed to feedAudioContent() and finishStream().
> feedAudioContent(args, kwargs)[source]
> Feed audio samples to an ongoing streaming inference.
> intermediateDecode(args, kwargs)[source]
> Compute the intermediate decoding of an ongoing streaming inference. This is an expensive process as the decoder implementation isn’t currently capable of streaming, so it always starts from the beginning of the audio.
Huh well that's a flaw the neglected to mention! I guess you can't really do what you want yet.