Why, yes. All life has the environmental constraints builtin in the structure of a brain and CNS.
Brain also has implicit sets of filters in sensory channels and in pattern-marching machinery.
Constraints and filters are the must have.
We can’t trust AI systems built on deep learning alone
11–20 of 133 posts
Re: We can’t trust AI systems built on deep learning alone
#12I started reading Rebooting AI last night. I think that Marcus and Davis (so far in the book) take a reasonable approach by wanting to design robust AI. Robust AI requires general real world intelligence that is not provided by deep learning. I have earned over 90% of my income over the last five or six years as a deep learning practitioner. I am a fan of DL based on great results for perception tasks as well as soli…
As much I'm hoping there'll be a breakthrough in AGI, maybe the right approach is the one AlphaGo was using: DL not as the top level decision-making, but plugged into a traditional decision-making algorithm in specific places.
Re: We can’t trust AI systems built on deep learning alone
#13[deleted]
Re: We can’t trust AI systems built on deep learning alone
#14I started reading Rebooting AI last night. I think that Marcus and Davis (so far in the book) take a reasonable approach by wanting to design robust AI. Robust AI requires general real world intelligence that is not provided by deep learning. I have earned over 90% of my income over the last five or six years as a deep learning practitioner. I am a fan of DL based on great results for perception tasks as well as soli…
As much I'm hoping there'll be a breakthrough in AGI, maybe the right approach is the one AlphaGo was using: DL not as the top level decision-making, but plugged into a traditional decision-making algorithm in specific places.
Yes. Though arguably this is just the same thing as the traditional approach to deep learning, which is to select features to input into a model. You don't always just have to train on raw information, pre-processing the input data and calculating specific features that we know, as humans, will be important for the final result and feeding those in addition to the raw data is a common approach. Don't see much difference between this and taking the output of a network and running it through a few decision trees. Most publishable projects applying AI you see typically have these type of human interaction on both sides of models.
Re: We can’t trust AI systems built on deep learning alone
#15The opposing argument, by Rich Sutton, Distinguished Research Scientist at DeepMind: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
[deleted]
Re: We can’t trust AI systems built on deep learning alone
#16I started reading Rebooting AI last night. I think that Marcus and Davis (so far in the book) take a reasonable approach by wanting to design robust AI. Robust AI requires general real world intelligence that is not provided by deep learning. I have earned over 90% of my income over the last five or six years as a deep learning practitioner. I am a fan of DL based on great results for perception tasks as well as soli…
As much I'm hoping there'll be a breakthrough in AGI, maybe the right approach is the one AlphaGo was using: DL not as the top level decision-making, but plugged into a traditional decision-making algorithm in specific places.
> As much I'm hoping there'll be a breakthrough in AGI
I think it probably won't be one breakthrough, but several, over decades. Personally, I'm pretty happy that AGI is taking a long time to materialize. We likely won't see a "fast takeoff scenario" (the computer is learning at a geometric rate !!1). It will likely happen gradually over years (progressively more intelligent, more aware computer systems), and we may have a chance to adapt in response.
Re: We can’t trust AI systems built on deep learning alone
#17The opposing argument, by Rich Sutton, Distinguished Research Scientist at DeepMind: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
there are some surprisingly weak arguments in the text. It's correct to not treat computational resources as constant, but ot treat them as unimportant or negligible is awful as well. Already computational resources are becoming prohibitive with only a few institutions producing state of the art models at high financial cost. If the goal is AGI this might get exponentially worse. Intelligence needs to take resource c…
> Already computational resources are becoming prohibitive with only a few institutions producing state of the art models at high financial cost.
That is something of an illusion.
Obviously there will be some sort of uneven distribution of computing power; some institutions will have more, some less. The institutions with more power will create models at the limit of what they can do, because that is the best use of their power.
So if the thesis of more power = more results holds then truly cutting results will always be by people with resources that are practically unattainable by everyone else. Google's AlphaGo wasn't a particularly clever model, for example. It just had a lot of horsepower behind it to train it and the various ranging shot attempts Deepmind would have gone through. Someone else would have figured it out albeit more slowly in a few years as computing power became available.
Computational power is still getting exponentially more affordable [0]. Costs aren't really rising, so much as the people who have spent more money get a few years ahead of everyone else and can preview what is about to become cheap.
[0] https://aiimpacts.org/recent-trend-in-the-cost-of-computing/
Re: We can’t trust AI systems built on deep learning alone
#18In this context, what is classical AI?
An SVM? A markov model? A large context free grammar with a dictionary?
Re: We can’t trust AI systems built on deep learning alone
#19Understanding a sentence is fundamentally different from recognizing an object. But people are trying to use deep learning to do both. I agree with most of the article but I think this^^ skips over the different types of networks used to solve perception and language problems. A CNN is very different from say, word2vec, which isn't a very deep network at all.
I’d go further and say that deep networks are excellent for sentence understanding, and various types of RNN or 1D convolutional layers are very good at this in specialized domains just as CNNs and ResNets are good in specialized vision applications.
It absolutely makes sense to use deep learning for both of these tasks.
In fact, one very effective thing to do is to use a Siamese network to learn joint representational spaces of text and imagery in the same network.
It’s really specious and disingenuous to say “boy, vision and language sure seem different but can you believe these DL researchers are using the same tools for both!?”
Re: We can’t trust AI systems built on deep learning alone
#20I've been waiting for the Symbolic/NN pendulum to starting swinging back the other way and start settling in the center. NN/DL is great for the interface between the outer world and the inner world of the mind (pattern recognition and re-construction), and symbolic AI more straightforwardly represents more "language of the mind" tasks, and easily handles issues like explanation and other meta-behaviors that with DL is difficult due to its black-box nature. DL's reliance on extension/training vs. intention/rules can develop ad-hoc intentional emergent theories which is their strength but also their weakness as these theories may not be correct or complete. Each can be brittle in their own way - so it'll be interesting to see more cross-pollination.