On a immediately useful practical note, OneNote also contains this functionality (obviously not as powerful). I've used it to record a meeting's audio sync'd to my notes, and then be able to search the audio to jump exactly to where someone mentioned something and review context. Saved my ass on at least one occasion.
Microsoft Research make breakthrough in audio speech recognition
31–40 of 46 posts
Re: Microsoft Research make breakthrough in audio speech recognition
#32For those keeping score, google's image feature extractor shares the same core principles as microsoft's speech recognizer. EDIT: by keeping score I mean keeping track of which techniques are being used where.
What are these core principles?
The main characters of both papers are many layered neural network architectures, autoencoders and stochastic gradient descent. The interesting thing is that all these ideas are from the 80s but the breakthrough was in how to use unsupervised learning to seed neural networks so that a many layered neural network did not get mired in local optima.
The key idea is that if you train each layer in an unsupervised manner and then feed its outputs as features for the next layer it performs better when you go on to train it in a supervised way. That is, back-propagation on the pre-trained Neural net, learns a far more robust set of weights than without pretraining. Stochastic gradient descent is a very simple technique that is useful for optimization when you are working with massive data.
The architecture Dahl used layers as RBM (very similar to autoencoders) to seed a regular ole but many layered Feedforward network. SGD is used to do back propagation. RBMs themselves are trained using a generative technique - see Contrastive divergence for more.
The google architecture is more complex and based on biological models. It is not trying to learn an explicit classifier hence they train a many layered autoencoder network to learn features. I only skimmed the paper but they have multiple layers specialized to a particular type of processing (think photoshop not intel) and using SGD they optimize an objective that is essentially learning an effective decomposition on the data.
The main takeaway is if you can find an effective way to build layered abstractions then you will learn robustly.
Re: Microsoft Research make breakthrough in audio speech recognition
#33How does this compare to Microsoft's Old HTK (HMM Toolkit)? The language used on the website seems to point to a lot of the same things. Is this breaking it down to actual IPA phonemes? I'm mostly curious because I used the HTK for my thesis and would like to know how they compare (besides, one being just 'newer').
This approach still uses HMMs, it's just that the observation probabilities are now coming from a DNN (neural network) instead of a GMM (gaussian mixture model). "Senones" are not new, HTK can use various context dependent phoneme models, and the HMM states (typically 3) within each context dependent phoneme essentially boil down to what they call a "senone" here. Interestingly, they use GMM's to bootstrap the DNN training -- which I suppose you could avoid once you have a reasonable DNN laying around.
The main difference here is hooking DNN output to an HMM decoder, replacing GMMs, and possibly even more important the training process they use to get the DNN fairly efficiently. That's the biggest thing -- GMMs, at least the last time I've looked, can be trained and adapted much quicker than a DNN.
Re: Microsoft Research make breakthrough in audio speech recognition
#34This seems very related to this http://www.youtube.com/watch?v=ZmNOAtZIgIk speak by Andrew Ng. It is a 40min speak, but he explains very simply how all this works for images and some examples about the audio case. It is incredible how using this deep learning techniques we can teach this "neural networks" to recognize such complicated patterns. It is like reverse engineering the brain's algorithms. BTW I took his Cou…
Are you still able to access the course materials? I took the course as well (and enjoyed it!) but I'd like to access the PDFs, especially.
Re: Microsoft Research make breakthrough in audio speech recognition
#35Earlier quoted context omitted.
Why alienate such a large segment of users after pouring so much money into their technology? The web is getting weirder. If a company invests in multiple markets, they should be prepared to do well in some markets and badly in others. Bing isn't as good as Google. Android isn't as well-designed as Metro. Yes, Android stole Apple's market, and, yes, Apple stole someone else's market. The large technology companies ar…
"Android stole Apple's market" eh? Must be why Apple is losing money hand over fist. Apple fans can be annoying but fandroids are often detached from reality.
Android market share is increasing at the expense of iPhone market share. That doesn't mean either of them are suffering. http://arstechnica.com/gadgets/2011/04/developer-frustration...
Re: Microsoft Research make breakthrough in audio speech recognition
#36Vlingo, Siri, and others have been doing speaker independent auto-adapting speech recognition for years and talking about systems requiring 'training' and improvements there sound like this article is 5 years old. Great to see innovation in this space but this article is very light on detail.
It is my understanding (albeit based on limited knowledge) that Siri, like other Nuance-powered systems that make a call to the server, are actually "trained" continuously by the huge amount of sample speech they receive by real users. The true "breakthrough" here would be if Microsoft made a voice recognition system that could run entirely on a device (no internet connection needed) and accurately understand speech…
As you might guess, they are not claiming this.
They basically are using a new (in the context of speech rec) technique that seems to improve accuracy by 16% relative on their test data (and using their code :-)). It's a really great result, but it doesn't change the basic nature of a state of the art speech recognizer at all -- you still need to train and adapt it -- and it still needs lots and lots of data.
Re: Microsoft Research make breakthrough in audio speech recognition
#37Earlier quoted context omitted.
Why alienate such a large segment of users after pouring so much money into their technology? The web is getting weirder. If a company invests in multiple markets, they should be prepared to do well in some markets and badly in others. Bing isn't as good as Google. Android isn't as well-designed as Metro. Yes, Android stole Apple's market, and, yes, Apple stole someone else's market. The large technology companies ar…
I think you are attributing to malice what is probably just laziness. It is fairly common for modern websites to drop the ball on support of some browser or other. I doubt Microsoft as a corporation made a deliberate decision to support IE and Firefox but not Chrome or Safari or Opera or whatever.
Chrome is more popular than Internet Explorer is!
Re: Microsoft Research make breakthrough in audio speech recognition
#38Earlier quoted context omitted.
Why alienate such a large segment of users after pouring so much money into their technology? The web is getting weirder. If a company invests in multiple markets, they should be prepared to do well in some markets and badly in others. Bing isn't as good as Google. Android isn't as well-designed as Metro. Yes, Android stole Apple's market, and, yes, Apple stole someone else's market. The large technology companies ar…
I think you are attributing to malice what is probably just laziness. It is fairly common for modern websites to drop the ball on support of some browser or other. I doubt Microsoft as a corporation made a deliberate decision to support IE and Firefox but not Chrome or Safari or Opera or whatever.
It's one thing to not test a site in a particular browser and to just put up an unobtrusive warning saying that some things might not work perfectly. It's quite another to actively block access based on the user agent string.
Re: Microsoft Research make breakthrough in audio speech recognition
#39For those keeping score, google's image feature extractor shares the same core principles as microsoft's speech recognizer. EDIT: by keeping score I mean keeping track of which techniques are being used where.
Do you mean the system behind the google images search?
Re: Microsoft Research make breakthrough in audio speech recognition
#40Earlier quoted context omitted.
What are these core principles?
The main characters of both papers are many layered neural network architectures, autoencoders and stochastic gradient descent. The interesting thing is that all these ideas are from the 80s but the breakthrough was in how to use unsupervised learning to seed neural networks so that a many layered neural network did not get mired in local optima. The key idea is that if you train each layer in an unsupervised manner…
RBM video presentation from 2007 by Hinton. Unsupervised learning is just amazing - imagine a neural net capable of "understanding" things it sees, but has never had direct instruction about. Just watch pictures, videos, let it "read" books and it learns. No human necessary.