Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploring Weight Agnostic Neural Networks

ai.googleblog.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Exploring Weight Agnostic Neural Networks

#51

Post by throwawaywego »

zwaps wrote: post #36
kolar wrote: post #35

How is this different from genetic programming?

How was early Machine Learning different from statistics? New names makes things exciting for people to oick up. Who wants to estimate multinomial regression when you can learn a shallow softmax activated neural network! Its all about creating hype.

> Statistical Modeling: The Two Cultures (2001), Breiman

> There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown. The statistical community has been committed to the almost exclusive use of data models. This commitment has led to irrelevant theory, questionable conclusions, and has kept statisticians from working on a large range of interesting current problems. Algorithmic modeling, both in theory and practice, has developed rapidly in fields outside statistics. It can be used both on large complex data sets and as a more accurate and informative alternative to data modeling on smaller data sets. If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools.

http://www2.math.uu.se/~thulin/mm/breiman.pdf

Re: Exploring Weight Agnostic Neural Networks

#52

Post by ma2rten »

elamje wrote: post #6

Unpopular quote from my image and video processing professor - “The only problem with machine learning is that the machine does the learning and you don’t.” While I understand that is missing a lot of nuance, it has stuck with me over the past few years as I feel like I am missing out on the cool machine learning work going on out there. There is a ton of learning about calculus, probability, and statistics when doin…

There has been a lot of research in understanding neural networks and making them less of a black box. If you classify cat from dog videos on YouTube, it doesn't matter if you make a mistake every now and again. But if you want to build a self-driving car or make a medical diagnosis, you better be able to explain which your network made a certain decision.

Re: Exploring Weight Agnostic Neural Networks

#53

Post by erikpukinskis »

ma2rten wrote: post #52
elamje wrote: post #6

Unpopular quote from my image and video processing professor - “The only problem with machine learning is that the machine does the learning and you don’t.” While I understand that is missing a lot of nuance, it has stuck with me over the past few years as I feel like I am missing out on the cool machine learning work going on out there. There is a ton of learning about calculus, probability, and statistics when doin…

There has been a lot of research in understanding neural networks and making them less of a black box. If you classify cat from dog videos on YouTube, it doesn't matter if you make a mistake every now and again. But if you want to build a self-driving car or make a medical diagnosis, you better be able to explain which your network made a certain decision.

Does the FDA require you are able to explain how drugs work? Or do you just have to show their efficacy and safety in trials?

Re: Exploring Weight Agnostic Neural Networks

#54

Post by bonoboTP »

ma2rten wrote: post #52
elamje wrote: post #6

Unpopular quote from my image and video processing professor - “The only problem with machine learning is that the machine does the learning and you don’t.” While I understand that is missing a lot of nuance, it has stuck with me over the past few years as I feel like I am missing out on the cool machine learning work going on out there. There is a ton of learning about calculus, probability, and statistics when doin…

There has been a lot of research in understanding neural networks and making them less of a black box. If you classify cat from dog videos on YouTube, it doesn't matter if you make a mistake every now and again. But if you want to build a self-driving car or make a medical diagnosis, you better be able to explain which your network made a certain decision.

I'm hearing this in the last few years quite often but I'm not sure what kind of explanation you mean.

The x-ray image was incorrectly misdiagnosed, because... What type of thing should come here?

... it didn't look like the other class. ... it didn't have this weird smudge thing on the top left in which case usually there should be a little hazier blob in the middle, except when the pointiness of the thing that is to the right of the brightest blabla...

You get the idea, my description is even exaggerating the nameability and describability of these structures. You'd get a long and complex description at best, because simple models don't work for pattern recognition. But even if you made sure to just use well understood features like edge thickness, angles, sizes of connected components etc, how would a boolean formula of a hundred such terms be helpful in court or wherever you want to use these explanations?

Re: Exploring Weight Agnostic Neural Networks

#55

Post by outlace »

kk58 wrote: post #50
outlace wrote: post #19

Earlier quoted context omitted.

I’m most excited about what is now being called scientific machine learning, ie machine learning models explicitly structured to learn interpretable models that can be used to understand some scientific domain better. For example, I’m starting to work on using graph RNNs to study dynamical behavior of reinforcement learning agents and how it might give us insights into psychiatric disorders.

I'm interested in reading more about this area of work. Can you share your project page if it exists or any foundational papers in this space

My inspiration comes from this: https://arxiv.org/pdf/1809.06303.pdf

There's a long history of using differential equation models to model high-level brain function (see also neural mass models and neural field models) but I think there are advantages to using discrete time approximations such as neural networks in an RL setting to investigate how the dynamics (e.g. attractor states, etc) map onto behavior.

Re: Exploring Weight Agnostic Neural Networks

#56

Post by jszymborski »

dna_polymerase wrote: post #47
amelius wrote: post #37

Earlier quoted context omitted.

ML is more like growing crops than it is about "designing stuff". Growing crops is slow, and you don't know beforehand what the result will be. However, you can still throw a lot of science at growing crops ("plant breeding" is a science), and the same holds for engineering.

Or you throw ML at growing crops, much like they throw ML at ML these days ( https://ai.googleblog.com/2017/05/using-machine-learning-to-... ).

This might be pedantry, but there are far sillier things to apply ML to than ML itself, despite the initial sound of the thing.

Re: Exploring Weight Agnostic Neural Networks

#57

Post by kuu »

pierrebai wrote: post #46
kuu wrote: post #22

Earlier quoted context omitted.

For me, the exciting part is to understand how the "blackbox" can learn and to find a representation of the data that makes this box to learn. For instance, I've been working in users profiling and it's been a challenge to find which features and in which representation allow the model to learn. It's fantastic when you make a little change in a feature (for instance, use the median instead of the mean) and your model…

"replace the mean with median" This illustrates the black-box aspect of it. You changed something and the results are affected but you don't know why. Median has a built-in implicit filtering (it's not affected by extreme outliers like the mean), so it could simply be that you needed to filter your inputs. But won't know, because... black box.

and the results are affected but you don't know why

Well, that's just your assumption without knowing the exact problem and I think you are missing my point.

You can approach to data preparation by randomly changing things, and maybe you can get some interesting results but I promise you you will fail many many times. Other way is to know what does it means to change the mean for the median for instance (as I mentioned in this just random example), and I promise you will find better solutions.

The idea is not just "change and test" and see what happens. The interesting part is to understand how the model uses your representation and why one is "better" than another.

Re: Exploring Weight Agnostic Neural Networks

#58

Post by TekMol »

patresh wrote: post #42
TekMol wrote: post #40

Is each architecture given one set of random weights? Or is the architecture of the net tested against a bunch of random weights so that it performs well independently of the weights?

Each architecture is tested multiple times against different samples of the shared weight From the paper : (1) An initial population of minimal neural network topologies is created (2) each network is evaluated over multiple rollouts, with a different shared weight value assigned at each rollout (3) networks are ranked according to their performance and complexity (4) a new population is created by varying the highes…

What is a "shared weight"?

Re: Exploring Weight Agnostic Neural Networks

#59

Post by TekMol »

reiinakano wrote: post #41
TekMol wrote: post #40

Is each architecture given one set of random weights? Or is the architecture of the net tested against a bunch of random weights so that it performs well independently of the weights?

Neither. The architecture performs well independently of weights BUT all weights must be the same e.g. Same network works well when all weights are 5.0 or when all weights are -3.0

Hmm... that collides with what patresh said.

Re: Exploring Weight Agnostic Neural Networks

#60

Post by reiinakano »

TekMol wrote: post #59
reiinakano wrote: post #41

Earlier quoted context omitted.

Neither. The architecture performs well independently of weights BUT all weights must be the same e.g. Same network works well when all weights are 5.0 or when all weights are -3.0

Hmm... that collides with what patresh said.

No, it doesn't. Shared weight means all weights are the same value, as I originally mentioned.
Post reply on HN