Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft Research make breakthrough in audio speech recognition

blogs.technet.com

Post reply on HN
Print view View original

41–46 of 46 posts

Re: Microsoft Research make breakthrough in audio speech recognition

#41

Post by tylerhobbs »

Dn_Ab wrote: post #32
mikedmiked wrote: post #22

Earlier quoted context omitted.

What are these core principles?

The main characters of both papers are many layered neural network architectures, autoencoders and stochastic gradient descent. The interesting thing is that all these ideas are from the 80s but the breakthrough was in how to use unsupervised learning to seed neural networks so that a many layered neural network did not get mired in local optima. The key idea is that if you train each layer in an unsupervised manner…

There's a very good Google Tech Talk by Geoff Hinton (who has worked closely with Dahl on a lot of this research and developed some of the key algorithms in this field) that explains how to build deep belief networks using layers of RBMs: http://www.youtube.com/watch?v=AyzOUbkUf3M

That video focuses on handwritten digit recognition, but it's great for understanding the basics. There's a second Google Tech Talk video from a few years later that talks directly about phoneme recognition as well: http://www.youtube.com/watch?v=VdIURAu1-aU

Re: Microsoft Research make breakthrough in audio speech recognition

#42

Post by gdahl »

bornhuetter wrote: post #2

Can someone please explain senones to me? Can't find much on Google. The article says that they are a fragment of a phoneme, but how small a fragment are we talking? 2-3 per phoneme, or many more? Also - I'd be curious how much the phoneme in a word can vary based on accent.

Senones are just tied triphone HMM states. A context dependent HMM recognizer has a 3-5 state HMM for every context dependent phone. Conceptually, each different HMM state in each different phone HMM has its own Gaussian mixture model, but this is awful because many of them don't get much data assigned to them. So people share parameters for different HMM states based on a data driven decision tree that clusters states together. Those clustered or tied states are sometimes called senones.

Re: Microsoft Research make breakthrough in audio speech recognition

#43

Post by gdahl »

acqq wrote: post #23

The most interesting bit for me is at the end of another blog entry: http://blogs.technet.com/b/inside_microsoft_research/archive... "An intern at Microsoft Research Redmond, George Dahl, now at the University of Toronto, http://www.cs.toronto.edu/~gdahl/ contributed insights into the working of DNNs and experience in training them. His work helped Yu and teammates produce a paper called Context-Dependent Pre-trained…

For people interested in some (currently) undocumented research code in python implementing DNNs that is also on my website. Although the code is only an initial release. I will improve it later, but if I waited until it wasn't embarrassing I would never release it, so I just posted it.

Re: Microsoft Research make breakthrough in audio speech recognition

#44

Post by gdahl »

breckinloggins wrote: post #14
cmicali wrote: post #8

Vlingo, Siri, and others have been doing speaker independent auto-adapting speech recognition for years and talking about systems requiring 'training' and improvements there sound like this article is 5 years old. Great to see innovation in this space but this article is very light on detail.

It is my understanding (albeit based on limited knowledge) that Siri, like other Nuance-powered systems that make a call to the server, are actually "trained" continuously by the huge amount of sample speech they receive by real users. The true "breakthrough" here would be if Microsoft made a voice recognition system that could run entirely on a device (no internet connection needed) and accurately understand speech…

The term "Deep Belief Network" has been abused in the literature (not pointing fingers, I've done it too). The DNNs used mean a neural net pre-trained with RBMs. Sometimes, when people say DBN, that is also what they mean. But really a DBN is a particular graphical model with undirected connections between the top two layers and directed connections everywhere else. The confusion comes from the pre-training procedure. The pre-training creates a DBN, which is then used to initialize the weights of a standard feedforward neural net. Then the DBN is discarded. It is a somewhat pedantic distinction. Since DBN is already an overloaded acronym (Dynamic Bayes Net) in the speech community and not entirely accurate for the pedantic reason I just mentioned, we decided to go with the DNN acronym.

Re: Microsoft Research make breakthrough in audio speech recognition

#45

Post by pathdependent »

gdahl wrote: post #43
acqq wrote: post #23

The most interesting bit for me is at the end of another blog entry: http://blogs.technet.com/b/inside_microsoft_research/archive... "An intern at Microsoft Research Redmond, George Dahl, now at the University of Toronto, http://www.cs.toronto.edu/~gdahl/ contributed insights into the working of DNNs and experience in training them. His work helped Yu and teammates produce a paper called Context-Dependent Pre-trained…

For people interested in some (currently) undocumented research code in python implementing DNNs that is also on my website. Although the code is only an initial release. I will improve it later, but if I waited until it wasn't embarrassing I would never release it, so I just posted it.

Thank you for doing so!

Overwhelmingly, it is my experience that researchers in computational disciplines publish papers with half-finished code "available on request" -- and requests are often ignored. It's refreshing to hear someone say, "Yes, the code needs work, but it should be available."

Re: Microsoft Research make breakthrough in audio speech recognition

#46

Post by brutuscat »

JabavuAdams wrote: post #34
brutuscat wrote: post #15

This seems very related to this http://www.youtube.com/watch?v=ZmNOAtZIgIk speak by Andrew Ng. It is a 40min speak, but he explains very simply how all this works for images and some examples about the audio case. It is incredible how using this deep learning techniques we can teach this "neural networks" to recognize such complicated patterns. It is like reverse engineering the brain's algorithms. BTW I took his Cou…

Are you still able to access the course materials? I took the course as well (and enjoyed it!) but I'd like to access the PDFs, especially.

Yes I have downloaded all the PDFs, email me (check my profile) I will share it via Dropbox for you ;)
Post reply on HN