One model to learn them all • Hacker News Board

Re: One model to learn them all

#31

Post by jiggunjer » Sat, Jan 13, 2018, 1:14 PM UTC

jfoutz wrote: post #22

erikpukinskis wrote: post #21

Legit question: if human intelligence is fundamentally limited by scale, why aren’t there some humans running around with heads twice as big and twice the neurons? If it’s such an advantage why hasn’t nature selected for it in at least one place on Earth? (My answer: above the scale of the human brain there’s diminishing returns on more neurons. Two individuals with their own volition are smarter than one individual…

Does smarter imply more children survive to reproduce?

I think it's more a plus for survival. But society has so much technology and excess resources that this pressure isn't affecting much.

Re: One model to learn them all

#32

Post by YeGoblynQueenne » Sat, Jan 13, 2018, 1:20 PM UTC

For me, as a non-practitioner (I'm happy with my GOFAI, thank you), the big problem with neural nets is that there are too many architectures, each tailored to a specific problem. On the one hand it's great that there's a broad toolset, on the other hand there are so many competing claims about best-of-class performance that it's hard to know what is even the state of the art. There is too much noise, you know?

So it'd be nice to see a result that reduced the noise a bit. I'm afraid this one doesn't fit the bill. It's not so much reducing architectural options, as piling even more architecture on top of the already sprawling mass of architecture. It's got architecture hanging from its architecture!

I mean, come on- one component is a gated-mixture-of-experts, which is to say, a collective of feed-forward nets. There are just so many layers upon layers of choices to make for each type of network to use for each component of the entire model. How do you make these choices?

Or, think about how long this specific architecture is going to give state-of-the art results (as claimed). It uses normalised ReLu convolution blocks- the height of fashion, at the moment, but what happens in four years from now, when nobody uses that anymore, because it's so 2010's?

ANN research frustrates me like this. It's describing an art form, but I'm not sure that's really the most useful thing to do.

Re: One model to learn them all

#33

Post by michaelmior » Sat, Jan 13, 2018, 1:37 PM UTC

YeGoblynQueenne wrote: post #32

For me, as a non-practitioner (I'm happy with my GOFAI, thank you), the big problem with neural nets is that there are too many architectures, each tailored to a specific problem. On the one hand it's great that there's a broad toolset, on the other hand there are so many competing claims about best-of-class performance that it's hard to know what is even the state of the art. There is too much noise, you know? So it…

I understand the frustration. I'm just getting into ANNs myself and the options are overwhelming. Trying to keep up with the state of the art is definitely a challenge. However, if you have a model that gives you the accuracy you need for whatever problem you're trying to solve, why not just stick with it? One of the nice things about using Neural nets these days is there's a lot of great software and infrastructure available for training and serving models.

Re: One model to learn them all

#34

Post by stewbrew » Sat, Jan 13, 2018, 2:43 PM UTC

erikpukinskis wrote: post #21

Legit question: if human intelligence is fundamentally limited by scale, why aren’t there some humans running around with heads twice as big and twice the neurons? If it’s such an advantage why hasn’t nature selected for it in at least one place on Earth? (My answer: above the scale of the human brain there’s diminishing returns on more neurons. Two individuals with their own volition are smarter than one individual…

> why aren’t there some humans running around with heads twice as big and twice the neurons?

Because humans don't live in an artificial bubble like ML models do but they always are already entangled in a network (aka society) of similar beings. The ultimate actor isn't the single individual but the group/aggregate/swarm/society. Given the current societal demands, the current brain size is sufficiently large. A single nerd with a double-sized brain would most likely not be able to reproduce in human societies as of today.

We probably shouldn't overrate a single individual's capabilities.

Re: One model to learn them all

#35

Post by Veedrac » Sat, Jan 13, 2018, 4:04 PM UTC

erikpukinskis wrote: post #21

Legit question: if human intelligence is fundamentally limited by scale, why aren’t there some humans running around with heads twice as big and twice the neurons? If it’s such an advantage why hasn’t nature selected for it in at least one place on Earth? (My answer: above the scale of the human brain there’s diminishing returns on more neurons. Two individuals with their own volition are smarter than one individual…

History seems to suggest precisely the opposite hypothesis; namely that the larger your brain is the more benefit there is from further growth, at least with this line of brain architecture.

http://aquatic-human-ancestor.org/anatomy/images/brain-size....

Re: One model to learn them all

#36

Post by tormeh » Sat, Jan 13, 2018, 4:09 PM UTC

joshmarlow wrote: post #24

erikpukinskis wrote: post #21

Legit question: if human intelligence is fundamentally limited by scale, why aren’t there some humans running around with heads twice as big and twice the neurons? If it’s such an advantage why hasn’t nature selected for it in at least one place on Earth? (My answer: above the scale of the human brain there’s diminishing returns on more neurons. Two individuals with their own volition are smarter than one individual…

Here's my understanding/interpretation - if I get any details wrong, I hope someone will correct me! Brains are very expensive and, for our body size, we have very large brains; evolution had to build a brain that could operate within the calorie constraints imposed by our ecological niche - presumably a hunter-gatherer niche. If you graph adult body volume relative to gestation time across the placental mammals, the…

Very interesting! I happen to know of some other problems: Heat and cross-brain communication time.

Bird brains are much smarter than we are per weight and volume. Not a little bit, but a lot. I think part of the reason they can do that is that they have much better cooling because their heads are small. Having a bigger brain than we do means that it would need to have a lower activity level because of cooling constraints.

There's also diminishing returns from size because the interconnects from one point in the brain to another have longer latency. One response to this is to fold the brain, as we do. Regardless, at some point adding more volume stops making sense.

Re: One model to learn them all

#37

Post by larister » Sat, Jan 13, 2018, 4:24 PM UTC

sdenton4 wrote: post #14

nicolashahn wrote: post #3

Earlier quoted context omitted.

To me it looks like they just took a bunch of specialized NN classifiers and glued them together. Not to belittle their work, this is still impressive and an important step towards generalized machine intelligence, but we're still a very long way off. The next level above this would be to give it some input and without telling the classifier what to do with it , it decides which task it's supposed to do on its own, a…

That actually seems like a rather higher bar than /we/ have to deal with. Image, auditory, touch and taste data come in on distinct signal paths before going to higher-level feature processing.

True in normal conditions, but our neocortex is highly adaptable. I can’t remember the studies, but there have been cases where a grid of actuators was taped to the body of a blind man and hooked up to a camera mounted on his head. In time, he was able to “see” edges based on the signals coming from his skin.

Re: One model to learn them all

#38

Post by marcosdumay » Sat, Jan 13, 2018, 4:26 PM UTC

erikpukinskis wrote: post #21

Legit question: if human intelligence is fundamentally limited by scale, why aren’t there some humans running around with heads twice as big and twice the neurons? If it’s such an advantage why hasn’t nature selected for it in at least one place on Earth? (My answer: above the scale of the human brain there’s diminishing returns on more neurons. Two individuals with their own volition are smarter than one individual…

With all the problems that big heads carry, human heads have done nothing but grow since our ancestors started walking on two feet.

I guess the only answer available for you is: evolution does not work that fast.

Re: One model to learn them all

#39

Post by ghthor » Sat, Jan 13, 2018, 4:45 PM UTC

YeGoblynQueenne wrote: post #32

For me, as a non-practitioner (I'm happy with my GOFAI, thank you), the big problem with neural nets is that there are too many architectures, each tailored to a specific problem. On the one hand it's great that there's a broad toolset, on the other hand there are so many competing claims about best-of-class performance that it's hard to know what is even the state of the art. There is too much noise, you know? So it…

You should look at the models being produced by Jeff Hawkins and numenta. Their neuron is based on the neuron of our neocortex which are homogeneous. Theyve had great success using their ANN across different domains with very little to no differences in settings. The magic really is in the encoding of the information stream into the Sparse Ditributed Representation that best represents the features you're looking for.

Re: One model to learn them all

#40

Post by mod » Sat, Jan 13, 2018, 5:06 PM UTC

larister wrote: post #37

sdenton4 wrote: post #14

Earlier quoted context omitted.

That actually seems like a rather higher bar than /we/ have to deal with. Image, auditory, touch and taste data come in on distinct signal paths before going to higher-level feature processing.

True in normal conditions, but our neocortex is highly adaptable. I can’t remember the studies, but there have been cases where a grid of actuators was taped to the body of a blind man and hooked up to a camera mounted on his head. In time, he was able to “see” edges based on the signals coming from his skin.

That's amazing.

I guess probably the orientation didn't even matter. So likely, you could seamlessly wear something like that on a patch of skin that stayed out of your way (on your back, perhaps).