Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning to Learn in TensorFlow

github.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Learning to Learn in TensorFlow

#11

Post by bike4beer »

rectangleboy wrote: post #2

Can someone explain to me the benefits of using TensorFlow over Theano?

https://indico.io/blog/the-good-bad-ugly-of-tensorflow/

Theano is better for the expert, TF is better for the beginner.

Long term planning is the key, TF is a modeling tool, but TF-Learn is better for testing architectures and algorithms.

Once your ready to go to a production system you would need to ditch all, and fall back to a solid environ that support C++ and distributed systems.

*

ML/DL by language/library

http://www.teglor.com/b/deep-learning-libraries-language-cm5...

Comparing Deep Language Software

https://en.m.wikipedia.org/wiki/Comparison_of_deep_learning_...

Note who support OPEN-MP, and who supports AMD GPU library's.

Long term most important consideration is how to you go to production.

Another problem with all this GOOGLE/AMAZON/MICROSOFT free ML/DL software is they all want to corral the users to a distributive system of running your algo's on THEIR servers for XXX $$/MIN, which means a taxi meter to run your own code; I think most people would prefer NOT to pay RENT to the NSA to research.

All these free frameworks are designed so that eventually people can 'automatically' have BIG-COMPANYS (NSA/CIA) keep their data & code and run it for them, which of course means in time little people may lose the ability to innovate and/or develop.

Re: Learning to Learn in TensorFlow

#12

Post by ericjang »

lucidrains wrote: post #6
panarky wrote: post #4

"Learning to learn by gradient descent by gradient descent" https://arxiv.org/abs/1606.04474

Other equally exciting papers that relates to learning to learn in DL. "Neural Architecture Search with Reinforcement Learning" https://arxiv.org/abs/1611.01578 "RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning" https://arxiv.org/abs/1611.02779 "Designing Neural Network Architectures using Reinforcement Learning" https://arxiv.org/abs/1611.02167

another one: Hyper-Networks https://arxiv.org/abs/1609.09106

and a blog post to go with it: http://blog.otoro.net/2016/09/28/hyper-networks/

Re: Learning to Learn in TensorFlow

#13

Post by lucidrains »

ericjang wrote: post #12
lucidrains wrote: post #6

Earlier quoted context omitted.

Other equally exciting papers that relates to learning to learn in DL. "Neural Architecture Search with Reinforcement Learning" https://arxiv.org/abs/1611.01578 "RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning" https://arxiv.org/abs/1611.02779 "Designing Neural Network Architectures using Reinforcement Learning" https://arxiv.org/abs/1611.02167

another one: Hyper-Networks https://arxiv.org/abs/1609.09106 and a blog post to go with it: http://blog.otoro.net/2016/09/28/hyper-networks/

This is why I love hackernews. Reading this tonight, thanks Eric :)

Re: Learning to Learn in TensorFlow

#14

Post by krab »

sja wrote: post #10
melse wrote: post #5

Earlier quoted context omitted.

I've heard that Tensorflow is built to take advantage of multiple GPUs automatically, whereas Theano (by default at least) can only make use of a single GPU.

"Automatically" isn't the best word, as TensorFlow won't make use of multiple GPUs unless you explicitly tell it to (at this time). That said, there are a number of benefits to using TensorFlow (including the ability to use multiple GPUs, if not automatically :) ) - Several common gradient optimization algorithms (Momentum, AdaGrad, AdaDelta, Adam, etc) are implemented already, which makes it a bit faster to get your…

> - There are tools for compiling your trained models as static C++ binaries on mobile devices

I'm looking for such tool but I haven't found anything apart from C++ libraries that also focus on training. Can you give me some pointers? Thanks.

Re: Learning to Learn in TensorFlow

#15

Post by oelmekki »

From the related article:

> The move from hand-designed features to learned features in machine learning has been wildly successful.

Are the features here the "feature vectors" or the network architecture? Or something else? In other terms, does this project help normalizing data, or does it help tweaking hyper parameters?

Re: Learning to Learn in TensorFlow

#16

Post by pepijndevos »

What's a good explanation of Tensor Flow for someone living under a rock? I dismissed it as some machine learning library, but I read it is in fact a general computing framework. If I can use it for things like numerical integration or some numpy-type tasks, that would be interesting.

Re: Learning to Learn in TensorFlow

#17

Post by dougabug »

oelmekki wrote: post #15

From the related article: > The move from hand-designed features to learned features in machine learning has been wildly successful. Are the features here the "feature vectors" or the network architecture? Or something else? In other terms, does this project help normalizing data, or does it help tweaking hyper parameters?

The point of the original paper was to learn the hyperparameters of a DNN using a DNN, as opposed to using, say, a bayesian optimization framework.

Re: Learning to Learn in TensorFlow

#18

Post by conmdur »

pepijndevos wrote: post #16

What's a good explanation of Tensor Flow for someone living under a rock? I dismissed it as some machine learning library, but I read it is in fact a general computing framework. If I can use it for things like numerical integration or some numpy-type tasks, that would be interesting.

Tensorflow does general computation using data flow graphs; you assemble your graph from operations and variables (tensors, as they're called these days), and Tensorflow handles distribution of this computation over hardware which you make available.

A quick google gives these [1] impressive results for Tensorflow, at least for linear algebra operations.

Despite the advantages, I think you'll find many more readily available functions in Numpy for what you want, while Tensorflow remains quite 'low level', exposing building block operations rather than higher-level methods (the exception is machine learning/neural network stuff). That said, I don't imagine it would be too difficult to implement a fast quadrature method for integration, or whatever else your heart might desire. This [2] is a simple example solving a PDE.

-----

[1] https://simplyml.com/linear-algebra-shootout-numpy-vs-theano...

[2] https://www.tensorflow.org/tutorials/pdes/

Re: Learning to Learn in TensorFlow

#19

Post by conmdur »

oelmekki wrote: post #15

From the related article: > The move from hand-designed features to learned features in machine learning has been wildly successful. Are the features here the "feature vectors" or the network architecture? Or something else? In other terms, does this project help normalizing data, or does it help tweaking hyper parameters?

Here the features are the feature vectors themselves, yes. It's been found that taking somewhat of a hands-off approach and allowing networks to engineer their own mid-level representations from raw data can be very beneficial.

This is the idea behind the learning to learn paper. Instead of taking our gradient and plugging it in to a hand-engineered (i.e. on paper) update rule, we feed it to a neural network, which is trained to find the optimal update rule, in some sense (neural networks are just function approximators after all).

Re: Learning to Learn in TensorFlow

#20

Post by spacehacker »

lucidrains wrote: post #6
panarky wrote: post #4

"Learning to learn by gradient descent by gradient descent" https://arxiv.org/abs/1606.04474

Other equally exciting papers that relates to learning to learn in DL. "Neural Architecture Search with Reinforcement Learning" https://arxiv.org/abs/1611.01578 "RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning" https://arxiv.org/abs/1611.02779 "Designing Neural Network Architectures using Reinforcement Learning" https://arxiv.org/abs/1611.02167

Another one:

'Learning to reinforcement learn'

https://arxiv.org/abs/1611.05763

Post reply on HN