Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploring Weight Agnostic Neural Networks

ai.googleblog.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: Exploring Weight Agnostic Neural Networks

#31

Post by elcomet »

DoctorOetker wrote: post #30
elcomet wrote: post #26

Earlier quoted context omitted.

I don't agree with your breakthrough. Reverse mode automatic différenciation is really very simple and is not at all what got us there. It's finding the right architectures that capture data priors and symmetries, and that can learn efficiently with stochastic gradient descent. Automatic differentiation is just a useful tool to use with those ideas.

I don't dispute the importance of architecture! But there was nothing new about architecture, essentially it is the choice of multidimensional function one tries to fit. In physics we have been fitting functions for hundreds of years. If you look at some experimental plot of say interference then you might decide to fit a sinusoid to it plus a background constant etc... the importance of fitting the right kind of fun…

I'm not sure I understand what you're saying. I think you're talking about gradient descent, not rm autograd. Gradient descent, which is a breakthrough dates back to the 19th century. That's what allow us to approximate gradients. RM autograd is a clever implement detail of this (how to compute gradients efficiently).

Re: Exploring Weight Agnostic Neural Networks

#32

Post by buboard »

DoctorOetker wrote: post #5

how is this different from boring old evolutionary algorithms? In my opinion the big breakthrough that enabled optimization and machine learning was the discovery of reverse mode automatic differentiation, since the space or family of all possible decision-functions is high dimensional, while the goal (survival, reproduction) is low dimensional. Unless I see a mathematical proof that evolutionary algorithms are as ef…

Evolutionary algorithms are perhaps easier to approach theoretically. Backprop works amazingly but how does one begin to approach why. There is also an element of backprop in evolution via epigenetics

Re: Exploring Weight Agnostic Neural Networks

#33

Post by nabla9 »

Y_Y wrote: post #27
nabla9 wrote: post #25

Earlier quoted context omitted.

Your professor was quoting/paraphrasing Alan Perlis in reverse. Epigram #63 63. When we write programs that "learn", it turns out we do and they don't. -- SIGPLAN Notices Vol. 17, No. 9, September 1982, pages 7 - 13.

Nice reference, but I think that claim is opposite to the professor's one.

you are correct.

Re: Exploring Weight Agnostic Neural Networks

#34

Post by nurettin »

https://github.com/google/brain-tokyo-workshop/tree/master/W... to me, this is the really interesting part of the article. NEAT (neuro-evolution of augmenting topologies) is an algorithm for GANN. For those who are looking to implement the algorithm from scratch, see http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf for hours of fun.

Re: Exploring Weight Agnostic Neural Networks

#36

Post by zwaps »

kolar wrote: post #35

How is this different from genetic programming?

How was early Machine Learning different from statistics?

New names makes things exciting for people to oick up. Who wants to estimate multinomial regression when you can learn a shallow softmax activated neural network!

Its all about creating hype.

Re: Exploring Weight Agnostic Neural Networks

#37

Post by amelius »

elamje wrote: post #6

Unpopular quote from my image and video processing professor - “The only problem with machine learning is that the machine does the learning and you don’t.” While I understand that is missing a lot of nuance, it has stuck with me over the past few years as I feel like I am missing out on the cool machine learning work going on out there. There is a ton of learning about calculus, probability, and statistics when doin…

ML is more like growing crops than it is about "designing stuff". Growing crops is slow, and you don't know beforehand what the result will be. However, you can still throw a lot of science at growing crops ("plant breeding" is a science), and the same holds for engineering.

Re: Exploring Weight Agnostic Neural Networks

#38

Post by choppaface »

kolar wrote: post #35

How is this different from genetic programming?

The authors draw from traditional genetic algorithms (NEAT http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf ) but this appears to be one of the first papers (at least recently) where proposed architectures are not trained but rather given random weights. The authors here try to qualitatively distill the role of architecture (vs optimization) in neural net research.

The work here is in a similar vein as the Lottery Ticket Hypothesis ( https://arxiv.org/pdf/1803.03635.pdf ), which found that deep nets (for vision) contain discriminative sub-networks at initialization time (due to random initialization), before training ever starts.

While the authors of this work on architecture search say they hope to inspire the discovery of new architectures, a more immediately striking result of their work is that they get functioning systems from doing something “stupid” (i.e. not optimizing weights).

Re: Exploring Weight Agnostic Neural Networks

#39

Post by YorkshireSeason »

zwaps wrote: post #36
kolar wrote: post #35

How is this different from genetic programming?

How was early Machine Learning different from statistics? New names makes things exciting for people to oick up. Who wants to estimate multinomial regression when you can learn a shallow softmax activated neural network! Its all about creating hype.

   How was early Machine Learning 
   different from statistics?
I'd argue: in two ways.

First: ML's algorithmic focus. Just about anything in modern AI/ML works because it uses compute at extreme scale. For example neural nets seem to work well only when trained with huge amounts of data. Statisticians lacked the background to make this happen.

Second: most work in statistics assumed that data was generated by given stochastic data model. In contrast, ML has been using algorithmic models and the data given by an unknown mechanism. In most real-world situations, the mechanism is unknown.

It's not just hype. Statistics was stuck in a local optimum, and it was ML's focus on algorithms, data structures, GPUs/TPUs, big data, ... together with the jump into 'weird' data (e.g. the proverbial cat photos), that propelled ML ahead of statistics.

Post reply on HN