71–80 of 164 posts
Very nice. How long before something this good is available as open source? A tough test would be to hook this up to a police/fire scanner, or air traffic control radio.
I don't have a background in this area, so I'm likely easily impressed, but this seems really impressive. And the acknowledgement that there's a lot of work to be done, such as discriminating between speakers and recognition in adverse environments. Yeah, it's Microsoft writing on their own technology, but they addressed in the text the questions I had already in mind from just reading the title. It didn't leave me w…
> It's frustrating when technologies like image and speech recognition and robotics are conflated with AI. Are you kidding? Of course these things are examples of artificial intelligence. I don't understand why people keep moving the goalposts wrt "AI".
My own taste uses the word "AI" as you do in a permissive way to include simpler tasks (which in themselves are more elemental than useful) like identifying an object in an image and presenting a few straightforward interpretations of sentences. But what Turing stipulated an actual AI would be able to do is "reach parity" with a fully human conversation, with all of our knowledge and values and desires and subtlety of motivation. When you really dig into what goes on in a real conversation---the joking, the shades of meaning, the individual quirks and tribal patterns, the lying, the compassion, ambition, insecurity---I think it's hard not to admit that we're still quite far from that. We may not even want it! But exactly the genius of the Turing test for setting a rough standard for AI is that it demands competences far beyond mere language processing.
The actual paper has a section on error analysis that is particularly enlightening: https://arxiv.org/abs/1610.05256 On the CallHome dataset humans confuse words 4.1% of the time, but delete 6.5% of words, most commonly deleting the word "I". Their ASR system confuses 6.5% of words on this dataset, but only deletes 3.3% of words, so depending on how you view this their claim about being better than humans isn't defin…
Earlier quoted context omitted.
They did post the code on github ( https://github.com/Microsoft/CNTK ) with the Microsoft open source license. Presumably you could feed it speech from a running instance of gnu-radio.
CNTK is just a toolkit, like tensorflow or theano. Code for the paper was not published.
The actual paper has a section on error analysis that is particularly enlightening: https://arxiv.org/abs/1610.05256 On the CallHome dataset humans confuse words 4.1% of the time, but delete 6.5% of words, most commonly deleting the word "I". Their ASR system confuses 6.5% of words on this dataset, but only deletes 3.3% of words, so depending on how you view this their claim about being better than humans isn't defin…
I'm not an expert on this at all, but my suspicion is that humans are more tolerant of missing words than we are of mistaken words. Especially a word like "I", we just assume it if it's missing. So if this speech recognition is for creating a transcript for humans, this (in my uneducated opinion) isn't as good as humans do, at least not yet.
The actual paper has a section on error analysis that is particularly enlightening: https://arxiv.org/abs/1610.05256 On the CallHome dataset humans confuse words 4.1% of the time, but delete 6.5% of words, most commonly deleting the word "I". Their ASR system confuses 6.5% of words on this dataset, but only deletes 3.3% of words, so depending on how you view this their claim about being better than humans isn't defin…
From the paper eight different recognizers combined together approached results of a single person reviewed once. Not really a parity.
The actual paper has a section on error analysis that is particularly enlightening: https://arxiv.org/abs/1610.05256 On the CallHome dataset humans confuse words 4.1% of the time, but delete 6.5% of words, most commonly deleting the word "I". Their ASR system confuses 6.5% of words on this dataset, but only deletes 3.3% of words, so depending on how you view this their claim about being better than humans isn't defin…
From the paper eight different recognizers combined together approached results of a single person reviewed once. Not really a parity.
Being Microsoft, I'm sure the place is diverse, but there's no mention in the paper on accents or dialects that I can see (might just be missing it).
The term "human parity" refers to a comparison of the error rate, which is a single scalar summarizing performance in terms of mistakes made. It says nothing about the kind of mistakes, and I can easily imagine that machines qualitatively do not make at all the same kind of mistakes as humans. I'd be curious to know if the kind of mistakes machines make might strike human listeners as quite stupid, but maybe not.. ma…
To explore that analogy: In the case of lossy audio compression, the compressor deliberately introduces quantization noise into the signal. It does so by running a "psychoacoustic model", which attempts to capture a broad quality of human hearing called auditory masking. There's a number of different kinds of masking[1]: a strong tonal sound creates an "umbrella" across nearby frequencies that can mask quieter noise-like sounds. Similarly there's noise-vs-noise masking, as well as forwards- and backwards- temporal masking. (Yes, backwards. A sound can mask perception of a sound that occurred before it.)
In the audio compression case, we've built an algorithm that attempts to characterize exploitable phenomena of human hearing. These masking characteristics aren't perceived quite the same by human listeners as the model, nor even the same between individual human listeners. Thus the need for human listening tests. These, due to the experimental care and human subjects required, are expensive.
Back to speech transcription. Say the end goal is "how well does a human comprehend this transcribed speech"? (e.g. vs some standard, such as the original speech, vs. the original speech transcribed by a skilled specialist, etc.) The problem starts to look pretty similar. We can cite numerical, word-centric error rates, but that fails to capture how well meaning is preserved and transmitted. Imagine a perverse algorithm that did a perfect transcription, but then dropped or altered words for maximum meaning obfuscation. It might equal or even beat the cited error rates but be much harder to actually comprehend.