How Google Translate squeezes deep learning onto a phone • Hacker News Board

Re: How Google Translate squeezes deep learning onto a phone

#11

Post by raverbashing » Wed, Jul 29, 2015, 4:01 PM UTC

joosters wrote: post #5

WordLens was an awesome app and it's good to see that Google is continuing the development. The new fad for using the 'deep' learning buzzword annoys me though. It seems so meaningless. What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' ?

> What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow'

Number of layers

It's that simple

Re: How Google Translate squeezes deep learning onto a phone

#12

Post by teraflop » Wed, Jul 29, 2015, 4:04 PM UTC

joosters wrote: post #5

WordLens was an awesome app and it's good to see that Google is continuing the development. The new fad for using the 'deep' learning buzzword annoys me though. It seems so meaningless. What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' ?

> What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' ?

If this is a serious question, then googling "what is a deep neural network" would take you to any number of explanations. But to summarize very briefly, it's not a buzzword; it's a technical term referring to a network with multiple nonlinear layers that are chained together in sequence. Deep networks have been talked about for as long as neural networks have been a research subject, but it's only in the last few years that the mathematical techniques and computational power have been available to do really interesting things with them.

The "fad" (as you call it) is not mainly because the word "deep" sounds cool, but because companies like Google have been seeing breakthrough results that are being used in production as we speak. For example:

http://papers.nips.cc/paper/4687-large-scale-distributed-dee...

http://static.googleusercontent.com/media/research.google.co...

Re: How Google Translate squeezes deep learning onto a phone

#13

Post by josu » Wed, Jul 29, 2015, 4:17 PM UTC

WordLens/Google Translate is the most futuristic thing that my phone is able to do. It's specially useful in countries that don't use the latin alphabet.

Re: How Google Translate squeezes deep learning onto a phone

#14

Post by joosters » Wed, Jul 29, 2015, 4:20 PM UTC

teraflop wrote: post #12

joosters wrote: post #5

WordLens was an awesome app and it's good to see that Google is continuing the development. The new fad for using the 'deep' learning buzzword annoys me though. It seems so meaningless. What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' ?

> What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' ? If this is a serious question, then googling "what is a deep neural network" would take you to any number of explanations. But to summarize very briefly, it's not a buzzword; it's a technical term referring to a network with multiple nonlinear layers that are chained together in sequence. Deep networks have been talked about fo…

I honestly didn't realise that it had any definition - I see now that calling it a 'fad' is unfair. However, the boundary between deep learning and (representational) machine learning still seems murky.

Re: How Google Translate squeezes deep learning onto a phone

#15

Post by motoboi » Wed, Jul 29, 2015, 4:20 PM UTC

I am 15 years into this computers thing and this blog post made me feel like "those guys are doing black magic".

Neural networks and deep learning are truly awesome technologies.

Re: How Google Translate squeezes deep learning onto a phone

#16

Post by ocdtrekkie » Wed, Jul 29, 2015, 4:24 PM UTC

teraflop wrote: post #12

joosters wrote: post #5

WordLens was an awesome app and it's good to see that Google is continuing the development. The new fad for using the 'deep' learning buzzword annoys me though. It seems so meaningless. What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' ?

> What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' ? If this is a serious question, then googling "what is a deep neural network" would take you to any number of explanations. But to summarize very briefly, it's not a buzzword; it's a technical term referring to a network with multiple nonlinear layers that are chained together in sequence. Deep networks have been talked about fo…

So was Word Lens doing this before Google even bought them? Because Word Lens worked fine, locally on a phone, long before Google was doing it's whole deep learning thing.

Re: How Google Translate squeezes deep learning onto a phone

#17

Post by dharma1 » Wed, Jul 29, 2015, 4:24 PM UTC

Would be great to see a more in depth article about this, and maybe even some open source code?

Re: How Google Translate squeezes deep learning onto a phone

#18

Post by dnautics » Wed, Jul 29, 2015, 4:26 PM UTC

raverbashing wrote: post #11

joosters wrote: post #5

WordLens was an awesome app and it's good to see that Google is continuing the development. The new fad for using the 'deep' learning buzzword annoys me though. It seems so meaningless. What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' ?

> What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' Number of layers It's that simple

It is that simple but the more complex story is that when the number of hidden layers exceeds 2, training becomes difficult. Also convnets for example cheat by having the connections between layers be incomplete bipartite graphs (not every node is connected to every other node), usually chosen because of some physical property - for computer vision nearest neighbors - eg.

Re: How Google Translate squeezes deep learning onto a phone

#19

Post by discardorama » Wed, Jul 29, 2015, 4:27 PM UTC

raverbashing wrote: post #11

joosters wrote: post #5

WordLens was an awesome app and it's good to see that Google is continuing the development. The new fad for using the 'deep' learning buzzword annoys me though. It seems so meaningless. What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' ?

> What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' Number of layers It's that simple

To expand on this some more: for a long time, thanks to Cybenko's theorem[1], people just used 1 hidden layer in their neural networks (also because computing was sloowww..). So, your typical NN architecture was input_layer --> hidden_layer --> output_layer.

Eventually, people realized that you could improve performance by adding more hidden layers. So while theoretically Cybenko was correct, practically stacking a bunch of hidden layers made more sense. These network architectures with stacks of hidden layers were then labelled as "deep" neural networks.

[1] https://en.wikipedia.org/wiki/Universal_approximation_theore...

Re: How Google Translate squeezes deep learning onto a phone

#20

Post by murbard2 » Wed, Jul 29, 2015, 4:28 PM UTC

I see no mention of it, but I'd be surprised if they didn't use some form of knowledge distilling [1] (which Hinton came up with, so really no excuse), to condense a large neural network into a much smaller one.

[1] http://arxiv.org/abs/1503.02531