Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speech Recognition Is Not Solved

awni.github.io

Post reply on HN
Print view View original

131–140 of 208 posts

Re: Speech Recognition Is Not Solved

#131

Post by Cacti »

dragontamer wrote: post #125
gok wrote: post #73

Earlier quoted context omitted.

The TPU architecture isn't that weird...it's basically a hardware implementation of matrix multiplication. It also isn't a silver bullet for ASR, where neural networks are usually only used for a part of the recognition process.

> The TPU architecture isn't that weird Its "weird" in the ways that matter: there's no commodity hardware in existence that replicates what a TPU does. The only place to get TPUs is through Google's cloud services. CPUs are basically Von Neumann Architecture. GPUs (NVidia and AMD) are basically SIMD / SIMT systems. Google's TPU is just something dramatically different, optimized yes for Matrix Multiplication, but it…

> but its not something you can buy and use offline.

But you will. The entire point is to put this in a phone, so you can distribute a trained neural net in a way that people can actually use without a desktop and $500-$4,000 GPU.

Re: Speech Recognition Is Not Solved

#132

Post by LandoCalrissian »

dharma1 wrote: post #107
LandoCalrissian wrote: post #79

Earlier quoted context omitted.

I recently have been trying out open source solutions for voice recognition for a personal project and you are very correct that it lags very far behind proprietary solutions. Pocketsphinx is still very limited and Kaldi takes quite a bit to setup in a usable fashion. There were a few other options I looked at that I can't think of from the top of my head but were all in similar condition. As the article says latency…

Kaldi is the best, there was just Tensorflow integration added which will hopefully speed up development (though I haven't seen any pretrained models for that yet). Here's a blog post - http://www.googblogs.com/kaldi-now-offers-tensorflow-integra... The easiest way to deploy Kaldi is this - https://github.com/alumae/kaldi-gstreamer-server (or a docker image of the that)

That's the option I was looking at using, so glad to see I'm on the right track. Thanks!

Re: Speech Recognition Is Not Solved

#133

Post by adrianbg »

jjwiseman wrote: post #128
adrianbg wrote: post #120

Earlier quoted context omitted.

The problem wrt open source / free solutions is data. Kaldi is open source and gets state of the art results -- but the data costs a lot of money. Training the models is doable on a commodity GPU although it takes quite a while.

I didn't realize Kaldi could get state of the art results. Do you say that because you know of people doing that, or is your comment based on knowing the architecture of Kaldi?

The 8.5% in this file is what you'd compare to Microsoft and IBM's recent ~5% results.

https://github.com/kaldi-asr/kaldi/blob/master/egs/fisher_sw...

Kaldi hasn't been in first place on that dataset recently, but it was a few years ago.

On other more researchy datasets (eg. for distant speakers or languages other than English), the best system is often based on Kaldi.

Re: Speech Recognition Is Not Solved

#134

Post by baddox »

jdmichal wrote: post #67
baddox wrote: post #45

Earlier quoted context omitted.

Problem definition really is the issue. Even appealing to the Turing test is less than satisfying. I’m fairly sure I’m a human, but Siri and similar will likely outperform me in certain categories like place names and popular music.

Don't forget that being too good can also cause you to fail the Turing test. I would not expect a human to be able to answer some questions that may be trivial to a computer. Things like, "What's the square-root of 137?" Or, "Identify this obscure song within 5 seconds of listening from a random starting point."

I suppose that depends on the precise setup of the test. Is the subject (if they're a human) allowed access to a calculator or a computer with an internet connection? Even if they were, timing would be an obvious tell, but an AI could easily be programmed (or could learn) to introduce an appropriate delay.

Re: Speech Recognition Is Not Solved

#135

Post by skywhopper »

canjobear wrote: post #10

It's always been a puzzle to me that published WER is so low, and yet when I use dictation (which is a lot--I use it for almost all text messages) I always have to make numerous corrections. This sentence is crucial and suggests a way to understand how WER can be apparently lower than human levels, yet ASR is still obviously imperfect: > When comparing models to humans, it’s important to check the nature of the mista…

Yes, exactly, the raw number of word errors is a very simplistic way to judge the accuracy of a transcription. Which words were incorrect and to what degree the failures changed the meaning are ultimately far more important. And while the test described in the article is a useful way to compare progress over time, it is clearly not nearly broad enough to cover the full range of scenarios humans will rightly expect automated speech recognition that "works" to be able to handle.

Re: Speech Recognition Is Not Solved

#136

Post by adrianbg »

dharma1 wrote: post #130
adrianbg wrote: post #124

Earlier quoted context omitted.

Exactly. This is also why Google sponsors the CHiME challenge, the existence of which is more proof that ASR is pretty far from solved. http://spandh.dcs.shef.ac.uk/chime_challenge/

Good stuff. Was looking into cheap array mics with linux drivers a few times in the past but not much is available. Speech separation - Mitsubishi Research has done some pretty impressive stuff on that - http://www.merl.com/demos/deep-clustering . Haven't seen equivalents of that in open source ASR

Totally. You may want to take a look at the papers from CHiME4 for more along those lines:

http://spandh.dcs.shef.ac.uk/chime_challenge/chime2016/resul...

I'm really fascinated by the whole idea of blind source separation and the fact that speech signals are "sparse" in frequency space.

We've had a similar experience looking for hardware / open source beamforming. There's a package called beamformit, but I think it's pretty old.

Re: Speech Recognition Is Not Solved

#137

Post by adrianbg »

skywhopper wrote: post #135
canjobear wrote: post #10

It's always been a puzzle to me that published WER is so low, and yet when I use dictation (which is a lot--I use it for almost all text messages) I always have to make numerous corrections. This sentence is crucial and suggests a way to understand how WER can be apparently lower than human levels, yet ASR is still obviously imperfect: > When comparing models to humans, it’s important to check the nature of the mista…

Yes, exactly, the raw number of word errors is a very simplistic way to judge the accuracy of a transcription. Which words were incorrect and to what degree the failures changed the meaning are ultimately far more important. And while the test described in the article is a useful way to compare progress over time, it is clearly not nearly broad enough to cover the full range of scenarios humans will rightly expect au…

I agree. I think a big part of the reason that people use WER is that it's relatively unambiguous and easy to measure.

Re: Speech Recognition Is Not Solved

#138

Post by StillBored »

Cacti wrote: post #131
dragontamer wrote: post #125

Earlier quoted context omitted.

> The TPU architecture isn't that weird Its "weird" in the ways that matter: there's no commodity hardware in existence that replicates what a TPU does. The only place to get TPUs is through Google's cloud services. CPUs are basically Von Neumann Architecture. GPUs (NVidia and AMD) are basically SIMD / SIMT systems. Google's TPU is just something dramatically different, optimized yes for Matrix Multiplication, but it…

> but its not something you can buy and use offline. But you will. The entire point is to put this in a phone, so you can distribute a trained neural net in a way that people can actually use without a desktop and $500-$4,000 GPU.

Is that the point? I ask, because the "weird" in the TPU is mostly its scale. Its not like you can't do matrix multiplies with the vector units on a CPU or with a GPU. Its really the scale, by that I mean its more elements than what you get with existing hardware, but its also lower precision, and appears less flexible, and is bolted to a heavyweight memory subsystem.

So, in that regard its no more "weird" than other common accelerator/coprocessors for things like compression.

So, in the end, what would show up in a phone doesn't really look anything like a TPU. I would maybe expect a lightweight piece of matrix acceleration hardware, which due to power constraints isn't going to be able to match what a "desktop" level FPGA or GPU is capable of much less a full blown TPU.

Re: Speech Recognition Is Not Solved

#139

Post by trengrj »

We recently did a comparative analysis of cloud speech-to-text providers for a project. We looked at:

1. Google Cloud Speech API

2. Microsoft Bing Speech API

3. IBM Watson Speech to Text

The ranking was as listed above but we had real challenges working with call-center audio recordings. The quality was less than idea but still very clear. We saw a huge reduction in accuracy compared to in-browser testing. Additionally Australian English is particularly not solved.

Because Google's API isn't currently doing speaker-detection, we looked at using Watson's speaker-detection as a secondary step but found it too complex and error prone. There is definitely room for a startup in this area and it also needs continued investment from the bigger cloud providers.

Re: Speech Recognition Is Not Solved

#140

Post by WalterBright »

> It’s the only way to go from ASR which works for some people, most of the time to ASR which works for all people, all of the time.

All people all the time doesn't remotely work for people, either. That's why the Air Force, for example, uses the Phonetic Alphabet, and aviation in general uses special jargon that is harder to misinterpret, such as "affirmative" for "yes".

http://militarytimechart.com/military-phonetic-alphabet/

Radio DJs and actors tend to enunciate more clearly than regular people do. You'll notice this more if you speed up the sound - the professionals are still clear at higher speeds where non-professionals become unintelligible.

Post reply on HN