Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Google Translate squeezes deep learning onto a phone

googleresearch.blogspot.com

Post reply on HN
Print view View original

91–99 of 99 posts

Re: How Google Translate squeezes deep learning onto a phone

#91

Post by deepnet »

deepnet wrote: post #80
thaumasiotes wrote: post #68

Earlier quoted context omitted.

The best Go bot approach (as of some years ago, but it's not like neural networks are a new idea) uses a very different strategy. Specifically, the strategy of "identify a few possible moves, simulate the game for several steps after each move using a very stupid move-making heuristic instead of using this actual strategy recursively , and then pick the move that yielded the best simulated board state".

Monte Carlo Tree Search ( Random playout ) is currently the best computer strategy for evaluating a Go position. This is likely due to the way Go works , random playout provides a rough estimate of who controls what territory ( this is how Go is scored ). Recently two deep-learning papers showed very impressive results. http://arxiv.org/abs/1412.3409 http://arxiv.org/abs/1412.6564 The neural networks were tasked with…

For Chess see David Silver's work on TreeStrap

It learns to master level from self-play.

http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Applications_fil...

also his lecture bootstrapping from tree based search

http://www.cse.unsw.edu.au/~cs9414/15s1/lect/1page/TreeStrap...

and Silver's overview on board game learning

http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching_files/g...

Re: How Google Translate squeezes deep learning onto a phone

#92

Post by deepnet »

zwegner wrote: post #85
Lawtonfogle wrote: post #36

Earlier quoted context omitted.

If there is magic to be found, it may be in that question. Why about graphs (namely the subset that are deep neural networks) allow them to not only contain such powerful heuristics, but also allow them to be created from scratch with barely any knowledge of the problem domain. As a side note, I was playing a board game last night (Terra Mystica I believe) and wondering if you could get 5 different neural networks to…

> Maybe Google could use this to make a Go bot. There was in fact a group within Google that worked on this: http://www.cs.toronto.edu/~cmaddis/pubs/deepgo.pdf

and the follow up from Google's Deepmind group :

Move Evaluation in Go Using Deep Convolutional Neural Networks Chris J. Maddison, Aja Huang, Ilya Sutskever, David Silver

http://arxiv.org/abs/1412.6564

Re: How Google Translate squeezes deep learning onto a phone

#93

Post by z92 »

deepnet wrote: post #92
zwegner wrote: post #85

Earlier quoted context omitted.

> Maybe Google could use this to make a Go bot. There was in fact a group within Google that worked on this: http://www.cs.toronto.edu/~cmaddis/pubs/deepgo.pdf

and the follow up from Google's Deepmind group : Move Evaluation in Go Using Deep Convolutional Neural Networks Chris J. Maddison, Aja Huang, Ilya Sutskever, David Silver http://arxiv.org/abs/1412.6564

Before clicking I was assuming it would fail. Then read this in the summary: "When the trained convolutional network was used directly to play games of Go, without any search, it beat the traditional search program GnuGo in 97% of games, and matched the performance of a state-of-the-art Monte-Carlo tree search that simulates a million positions per move."

Re: How Google Translate squeezes deep learning onto a phone

#94

Post by pedrosorio »

megalodon wrote: post #90
liabru wrote: post #89

Earlier quoted context omitted.

Could you link to it please? I've not tried it on anything else, but I remember thinking that it has a lot of potential uses. Also I only used it on gray-scale features, but I'm sure it could make use of full RGB too. I'll have to try it some time!

"We also investigated hierarchical features where the image is overlaid with a grid of cell size c × c and pixels withins each cell are added up. This is same as downsampling the image and using the raw pixels in the downsampled image as features." (p. 3) http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-15...

Sounds similar to one level of a pyramid:

https://en.wikipedia.org/wiki/Pyramid_(image_processing)

Re: How Google Translate squeezes deep learning onto a phone

#95

Post by alok-g »

pschanely wrote: post #72

Doesn't this article seem to say that the size of the training set is related to the size of the resulting network? It should be proportional to the number of nodes/layers that the network is configured for, not proportional to the number of training instances. Am I missing something?

The network is sized to be able to learn the training data reasonably well (e.g. via hyper-parameter optimization). If there is too much variation in data that is not seen in the real application (like rotation of letters mentioned in the article), an appropriately sized network will still learn it, but would be an overkill for the application at hand.

Re: How Google Translate squeezes deep learning onto a phone

#96

Post by lisivka »

dnautics wrote: post #18
raverbashing wrote: post #11

Earlier quoted context omitted.

> What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow' Number of layers It's that simple

It is that simple but the more complex story is that when the number of hidden layers exceeds 2, training becomes difficult. Also convnets for example cheat by having the connections between layers be incomplete bipartite graphs (not every node is connected to every other node), usually chosen because of some physical property - for computer vision nearest neighbors - eg.

Use another deep learning network to supervise training of your DLN. You can also use it to supervise itself. It is simple idea invented about decade ago (at least I heard it about decade ago here, in Ukraine).

Re: How Google Translate squeezes deep learning onto a phone

#97

Post by birdsbolt »

Why do they need a deep learning model for this? They are obviously targeting signs, product names, menus and similar. Model will obviously fail in translating large texts.

Was there any advantage of using a deep learning model instead of something more computationally simple?

Re: How Google Translate squeezes deep learning onto a phone

#98

Post by anantzoid »

sp332 wrote: post #50
anantzoid wrote: post #32

Just waiting for the paper to come out that'll detail all the transformations that were done on the training data specifically for the phone and how did they arrive at deciding to use them. > To achieve real-time, we also heavily optimized and hand-tuned the math operations. That meant using the mobile processor’s SIMD instructions and tuning things like matrix multiplies to fit processing into all levels of cache me…

Not fitting into cache just means it will run slower. Why would it crash?

Other apps getting slow is also not a very good thing!

Re: How Google Translate squeezes deep learning onto a phone

#99

Post by mistercow »

dr_zoidberg wrote: post #28
motoboi wrote: post #15

I am 15 years into this computers thing and this blog post made me feel like "those guys are doing black magic". Neural networks and deep learning are truly awesome technologies.

They are, but once you start learning about them, you realize the "black magic" part comes mostly from their mathematical nature and very little from them being "inteligent computers". A neural net is a graph, in which a subset of nodes are "inputs" (that's where the net gets information), some are outputs, and there are other nodes which are called "hidden neurons". The nodes are interconnected between each other in…

> Deep learning means ... which apply convolution operators

Convolutional networks are only one kind of deep learning. In particular, they generally apply only to image processing.

Post reply on HN