This reminds me of an oldie, but goodie: http://thedailywtf.com/articles/No%2C_We_Need_a_Neural_Netwo...
No Quack.
When to assume neural networks can solve a problem
31–40 of 44 posts
Re: When to assume neural networks can solve a problem
#32it seems that neural networks have problem for computing the maximum function, and a human can compute the maximum easily, so it seems that the three heuristic rules don't work in this case. (1) https://datascience.stackexchange.com/questions/56676/can-ma...
That is untrue, Here's a code example (actually took me ~20 minutes to get it "right" so I'll admit it's not the most trivial problem)... it includes seeds so that you can replicate locally (it should hit 100% accuracy all the time on the 1200 examples testing set reliably by about epoch 700): ''' import torch import random from sklearn.metrics import accuracy_score random.seed(61) torch.manual_seed(61) X = [[random.…
You are right. Also the Universal Approximation theorem (1) for neural networks guarantees that neural networks can approximate continuous function on compact subsets of R^n, in this case max(x,y).
argmax([x,y]) = (sign(x[0]-x[1])+1)/2
Going beyond continuous functions, can deep learning be used for primality test?
(1) https://en.wikipedia.org/wiki/Universal_approximation_theore...
Re: When to assume neural networks can solve a problem
#33I've been hearing about deep learning revolutionizing "everything" for the past 8 years. So, can someone name me any significant impact it made on figuring out what to do about the recent epidemic? If not, I think it's worth reflecting on what value we get out of the technology that sucked up so much of our intellectual, financial and computational resources.
It's not revolutionizing everything in the field of AI. It's "only" good at prediction in domains where data is poorly structured, such as images, audio, video, text and some games (which is a lot to be fair, hence the publicity it got in the recent years). It's not that great on structured data or on small datasets, and is not really adapted to anything not related to raw, "black-boxed" prediction.
I'm not sure what you'd expect that tool to do regarding the current epidemic.
Re: When to assume neural networks can solve a problem
#34I've made a neural network where you feed it a problem and it will tell you if a neural network will solve it. I call it a Neural decider. Getting weird results when I feed it to itself though.
Neuroboros?
Re: When to assume neural networks can solve a problem
#35it seems that neural networks have problem for computing the maximum function, and a human can compute the maximum easily, so it seems that the three heuristic rules don't work in this case. (1) https://datascience.stackexchange.com/questions/56676/can-ma...
You're referring to about whether a generic one-size-fits-all model will do well, but ML is full of bespoke models. It would be simple to build a neural network that can compute (and differentiate through) the max function to within some arbitrary epsilon, even though the most generic model (feed forward network) won't do great.
Re: When to assume neural networks can solve a problem
#36> 3. A neural network can solve problems that a human can solve with small-sized datapoints and little to no context
I like this rule, because it is actionable. Its easy to take a sample of your data, (in the case of images) lower the quality to 20x20 pixels and get see if you, or another person can do the classification task. Audio could be handled similarly.
Re: When to assume neural networks can solve a problem
#37it seems that neural networks have problem for computing the maximum function, and a human can compute the maximum easily, so it seems that the three heuristic rules don't work in this case. (1) https://datascience.stackexchange.com/questions/56676/can-ma...
You're referring to about whether a generic one-size-fits-all model will do well, but ML is full of bespoke models. It would be simple to build a neural network that can compute (and differentiate through) the max function to within some arbitrary epsilon, even though the most generic model (feed forward network) won't do great.
> l (feed forward network) won't do great.
See my answer below, in the case of this problem a generic feed-forward network, even a simple one, will work.
Not any ffn, but assuming you are using an efficient architecture search it will probably find one that works.
There's other numerical problems where this doesn't hold but that's another story.
Re: When to assume neural networks can solve a problem
#38Earlier quoted context omitted.
That is untrue, Here's a code example (actually took me ~20 minutes to get it "right" so I'll admit it's not the most trivial problem)... it includes seeds so that you can replicate locally (it should hit 100% accuracy all the time on the 1200 examples testing set reliably by about epoch 700): ''' import torch import random from sklearn.metrics import accuracy_score random.seed(61) torch.manual_seed(61) X = [[random.…
You are right. Also the Universal Approximation theorem (1) for neural networks guarantees that neural networks can approximate continuous function on compact subsets of R^n, in this case max(x,y). argmax([x,y]) = (sign(x[0]-x[1])+1)/2 Going beyond continuous functions, can deep learning be used for primality test? (1) https://en.wikipedia.org/wiki/Universal_approximation_theore...
https://escholarship.org/content/qt5sg7n4ww/qt5sg7n4ww.pdf
> A long-standing difficulty for connectionism has been to implement compositionality, the idea of building a knowledge representation out of components such that the meaning arises from the meanings of the individual components and how they are combined. Here we show how a neural-learning algorithm, knowledge-based cascade-correlation (KBCC), creates a compositional representation of the prime-number concept and uses this representation to decide whether its input n is a prime number or not. KBCC conformed to a basic prime-number testing algorithm by recruiting source networks representing division by prime numbers in order from smallest to largest prime divisor up to √n. KBCC learned how to test prime numbers faster and generalized better to untrained numbers than did similar knowledge-free neural learners. The results demonstrate that neural networks can learn to perform in a compositional manner and underscore the importance of basing learning on existing knowledge.
But again, I think things such as prime number tests are the exact kind of edge cases where one needs too many heuristics built into the model for it to be practical to use.
But I think something like a prime test is not included under the definition I gave anyway, because the idea of "prime" actually implies a lot of context.
You can take a baby and he will be able to classify images, you can take a human that speaks a language with no concept of numbers and he will be able to play or sing music and distinguish patterns in it.
You can't talk about "prime" without a mathematical apparatus that takes years for humans to understand. However, since we learn it as such an early age, it ends up in the background.
Granted, that could be said about almost any cognitive ability (the fact that there's a lot of "subconscious context" required to use it).... so I don't know.
Re: When to assume neural networks can solve a problem
#39Earlier quoted context omitted.
You are right. Also the Universal Approximation theorem (1) for neural networks guarantees that neural networks can approximate continuous function on compact subsets of R^n, in this case max(x,y). argmax([x,y]) = (sign(x[0]-x[1])+1)/2 Going beyond continuous functions, can deep learning be used for primality test? (1) https://en.wikipedia.org/wiki/Universal_approximation_theore...
https://escholarship.org/content/qt5sg7n4ww/qt5sg7n4ww.pdf > A long-standing difficulty for connectionism has been to implement compositionality, the idea of building a knowledge representation out of components such that the meaning arises from the meanings of the individual components and how they are combined. Here we show how a neural-learning algorithm, knowledge-based cascade-correlation (KBCC), creates a compo…
[deleted]
Re: When to assume neural networks can solve a problem
#40Earlier quoted context omitted.
You're referring to about whether a generic one-size-fits-all model will do well, but ML is full of bespoke models. It would be simple to build a neural network that can compute (and differentiate through) the max function to within some arbitrary epsilon, even though the most generic model (feed forward network) won't do great.
> l (feed forward network) won't do great. See my answer below, in the case of this problem a generic feed-forward network, even a simple one, will work. Not any ffn, but assuming you are using an efficient architecture search it will probably find one that works. There's other numerical problems where this doesn't hold but that's another story.
You demonstrated it for a reeeeeeeallly constrained version of the problem. Do you expect your solution would generalize to many lists? Because it would be easy to make a neural network that does, while your toy example (and larger generalizations) probably won't generalize super well.
x_i = ith list element from list x
y = sum(x_i * softmax(k * x)_i)
This one parameter, arbitrarily wide network one will get arbitrarily close to the max function.
This is a super toy version of why attention is so effective. It can pick stuff.