Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSpeech 0.6

hacks.mozilla.org

Post reply on HN
Print view View original

41–50 of 68 posts

Re: DeepSpeech 0.6

#41

Post by IshKebab »

reubenmorais wrote: post #11
detaro wrote: post #5

> It achieves a 7.5% word error rate on the LibriSpeech test clean benchmark Anyone have a comparison for how good/bad that is compared to other solutions, and what it means for practical usage, if that can be guessed at from a single number?

Author here. I would add to what the sibling comments have mentioned by saying that SotA results should be taken with a grain of salt. Our engine is capable of streaming (processing the audio as it's being recorded), which is not doable with architectures that have bidirectional decoders or attention mechanisms that require the whole encoder input ahead of time. For real world applications, this is absolutely crucial…

> Our engine is capable of streaming

A bit misleading given that decoding can't be streamed yet.

Re: DeepSpeech 0.6

#42

Post by IshKebab »

reubenmorais wrote: post #11
detaro wrote: post #5

> It achieves a 7.5% word error rate on the LibriSpeech test clean benchmark Anyone have a comparison for how good/bad that is compared to other solutions, and what it means for practical usage, if that can be guessed at from a single number?

Author here. I would add to what the sibling comments have mentioned by saying that SotA results should be taken with a grain of salt. Our engine is capable of streaming (processing the audio as it's being recorded), which is not doable with architectures that have bidirectional decoders or attention mechanisms that require the whole encoder input ahead of time. For real world applications, this is absolutely crucial…

> For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds.

You should follow Google's approach - give fast live results that don't depend on data from the future, but also go back and correct old words when you do have that data. It's kind of how humans work really.

Re: DeepSpeech 0.6

#43

Post by mncharity »

reubenmorais wrote: post #28
na85 wrote: post #25

Earlier quoted context omitted.

>Just use it as is and see how it does. How? The docs seem to be suffering from the same problem as the git docs: they assume the reader is already a domain expert, which would seem to run counter to the stated goal of being simple (and ubiquitous). I tried to figure out how to "just use" deepspeech and failed.

There's a basic example here transcribing from an audio file: https://github.com/mozilla/DeepSpeech/tree/v0.6.0/#project-d... There are also examples transcribing from the microphone as I mentioned above: https://github.com/mozilla/DeepSpeech/tree/master/examples/ I also have a small GUI example here [0], click once to start recording, once again to stop and show transcript. It receives the same command line argument…

> There are also examples

Fyi, the other 4 example links there are 404. Also in the root README. The linked README's are missing, or diversely suffixed and mislinked.

Re: DeepSpeech 0.6

#44

Post by lunixbochs »

reubenmorais wrote: post #11
detaro wrote: post #5

> It achieves a 7.5% word error rate on the LibriSpeech test clean benchmark Anyone have a comparison for how good/bad that is compared to other solutions, and what it means for practical usage, if that can be guessed at from a single number?

Author here. I would add to what the sibling comments have mentioned by saying that SotA results should be taken with a grain of salt. Our engine is capable of streaming (processing the audio as it's being recorded), which is not doable with architectures that have bidirectional decoders or attention mechanisms that require the whole encoder input ahead of time. For real world applications, this is absolutely crucial…

> For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds.

My wav2letter research does not use streaming, despite supporting streaming, because I get lower average latency (roughly 0.02x RTF including decoding, or 80ms for a 4000ms input audio like you describe in the article) by running the entire encoder CNN at once instead of running it in chunks.

This is also nice because the CPU usage is practically nothing (0.1%) until you stop talking.

I have a web demo here [1], with a relatively terrible language model (mostly just been working on acoustic modeling so far, as it’s just me). Most of the latency is waiting for the JavaScript VAD (which is much slower than webrtcvad and I couldn’t figure out how to tune it) and waiting for the network. If you look at the network inspector, the server should report its encode and decode times.

[1] https://web2letter-west-1.talonvoice.com/

Re: DeepSpeech 0.6

#45

Post by lunixbochs »

IshKebab wrote: post #41
reubenmorais wrote: post #11

Earlier quoted context omitted.

Author here. I would add to what the sibling comments have mentioned by saying that SotA results should be taken with a grain of salt. Our engine is capable of streaming (processing the audio as it's being recorded), which is not doable with architectures that have bidirectional decoders or attention mechanisms that require the whole encoder input ahead of time. For real world applications, this is absolutely crucial…

> Our engine is capable of streaming A bit misleading given that decoding can't be streamed yet.

I’m not sure what you mean. The image in the blog post clearly shows small chunks of decode time happening during the speech (every second).

Re: DeepSpeech 0.6

#46

Post by lunixbochs »

Traktor wrote: post #6
detaro wrote: post #5

> It achieves a 7.5% word error rate on the LibriSpeech test clean benchmark Anyone have a comparison for how good/bad that is compared to other solutions, and what it means for practical usage, if that can be guessed at from a single number?

You can find a leaderboard here: https://paperswithcode.com/sota/speech-recognition-on-libris... It seems like sota is 2.20% word error rate

I like reading through https://github.com/syhw/wer_are_we sometimes

Re: DeepSpeech 0.6

#47

Post by candiodari »

bayesian_horse wrote: post #7

I wonder if it is any good at out-of-vocabulary words? Is it hard to teach it new things like medical terms and such?

It's using letters, not words, so in theory anything that can be spelled in English can be output:

https://2r4s9p1yi1fa2jd7j43zph8r-wpengine.netdna-ssl.com/fil...

Or, if you prefer the code: https://github.com/mozilla/DeepSpeech/blob/0427c1572ac8f7253...

(well, I suppose technically it's in https://github.com/mozilla/DeepSpeech/blob/master/data/alpha... through https://github.com/mozilla/DeepSpeech/blob/master/util/text...., but I mean come on)

In practice ... meh. It gets some things predictably wrong. They're readable, just wrong. So the good news is that WER is words it has 100% correct. The remaining errors aren't usually totally unreadable, just irritating, going from "totext" instead of "to text", but of course names go pretty wrong and don't get better. It butchers my name, of course.

If you test it yourself, please do realize it really doesn't deal very well with accents.

Re: DeepSpeech 0.6

#48

Post by Enginerrrd »

jcims wrote: post #36

This reminds me of something I would love to see happen but I don't have the skills to put it all together. I really think there's some potential merit to a reading coach app(lication) that listens to someone read and looks for weaknesses/disorders/etc compared to a trained model. It could provide those diagnostics to an educator, guide the content to focus on those, coach the reader directly, etc. It all seems very…

I'll be honest... that seems like a pretty tough sell to me. You could be looking at anything from a speech disorder to a learning disability to a perfectly healthy normal child with idiosyncrasies and those things are usually left to professionals with graduate degrees. The liability seems high, you've got both the potential stigma associated with incorrectly flagging a child, or worse, missing an otherwise obvious issue that means treatment is delayed during critical years.

Re: DeepSpeech 0.6

#49

Post by sgt101 »

mncharity wrote: post #43
reubenmorais wrote: post #28

Earlier quoted context omitted.

There's a basic example here transcribing from an audio file: https://github.com/mozilla/DeepSpeech/tree/v0.6.0/#project-d... There are also examples transcribing from the microphone as I mentioned above: https://github.com/mozilla/DeepSpeech/tree/master/examples/ I also have a small GUI example here [0], click once to start recording, once again to stop and show transcript. It receives the same command line argument…

> There are also examples Fyi, the other 4 example links there are 404. Also in the root README. The linked README's are missing, or diversely suffixed and mislinked.

You are right, and I expect Reuben and team will fix this and are grateful for the heads up.

I was able to find the right links though, for example the WAV transcriber is at : https://github.com/mozilla/DeepSpeech/blob/v0.6.0/examples/v...

Re: DeepSpeech 0.6

#50

Post by IshKebab »

lunixbochs wrote: post #45
IshKebab wrote: post #41

Earlier quoted context omitted.

> Our engine is capable of streaming A bit misleading given that decoding can't be streamed yet.

I’m not sure what you mean. The image in the blog post clearly shows small chunks of decode time happening during the speech (every second).

The API documentation says decoding starts from the beginning every time. Could just be the Python API hasn't been updated I guess?

Edit: never mind - the API documentation was just out of date.

Post reply on HN