Artificial Intelligence Software Is Booming, But Why Now?
1–10 of 141 posts
Re: Artificial Intelligence Software Is Booming, But Why Now?
#2Because it's been just about 10 years since they figured out how to:
1) Use convolutional layers, ReLUs, and a few other tricks to ameliorate the vanishing gradient problem,
2) Perform continuous, high-dimensional stochastic gradient descent on graphics cards, and
3) Apply these things via stochastic grad-student descent to sufficiently massive datasets that even the most brute-force models and training methods (backpropagation of errors on a loss functional) can actually work.
In the meantime, the hardware for doing it has become commoditized and the software has consolidated and become standardized. So now it's A Thing in industry.
There are lots of "smarter" algorithms that almost definitely come closer to human cognition, for instance probabilistic program induction. But those aren't fast, and don't always neatly separate training from prediction: you're just not gonna be able to train those models ahead-of-time on a 10,000 image corpus inside a single week with today's hardware and software.
We need to find ways to make machine learning fast even when it's not just a bunch of matrix multiplies. Otherwise, every time we make our algorithms more interesting, we cripple ourselves computationally.
Re: Artificial Intelligence Software Is Booming, But Why Now?
#3Because it's been just about 10 years since they figured out how to: 1) Use convolutional layers, ReLUs, and a few other tricks to ameliorate the vanishing gradient problem, 2) Perform continuous, high-dimensional stochastic gradient descent on graphics cards, and 3) Apply these things via stochastic grad-student descent to sufficiently massive datasets that even the most brute-force models and training methods (back…
> There are lots of "smarter" algorithms that almost definitely come closer to human cognition
Source?
Re: Artificial Intelligence Software Is Booming, But Why Now?
#4Because it's been just about 10 years since they figured out how to: 1) Use convolutional layers, ReLUs, and a few other tricks to ameliorate the vanishing gradient problem, 2) Perform continuous, high-dimensional stochastic gradient descent on graphics cards, and 3) Apply these things via stochastic grad-student descent to sufficiently massive datasets that even the most brute-force models and training methods (back…
I'm concerned about something similar, where a lot of AI techniques seem to be rushing toward a local maximum:
* AI researchers do things that get good results faster out of Nvidia cards
* Nvidia makes their cards faster at the things AI researchers are doing
It's getting good results. We sure are going up this gradient quickly. But I don't think it's going to get us to a global maximum.
Re: Artificial Intelligence Software Is Booming, But Why Now?
#5Because it's been just about 10 years since they figured out how to: 1) Use convolutional layers, ReLUs, and a few other tricks to ameliorate the vanishing gradient problem, 2) Perform continuous, high-dimensional stochastic gradient descent on graphics cards, and 3) Apply these things via stochastic grad-student descent to sufficiently massive datasets that even the most brute-force models and training methods (back…
4) Frameworks and libraries have improved a lot and are available as open source. Having a lower barrier of entry helps.
Re: Artificial Intelligence Software Is Booming, But Why Now?
#6Because it's been just about 10 years since they figured out how to: 1) Use convolutional layers, ReLUs, and a few other tricks to ameliorate the vanishing gradient problem, 2) Perform continuous, high-dimensional stochastic gradient descent on graphics cards, and 3) Apply these things via stochastic grad-student descent to sufficiently massive datasets that even the most brute-force models and training methods (back…
I'm definitely going to use that stochastic grad student descent line.
Re: Artificial Intelligence Software Is Booming, But Why Now?
#7Because it's been just about 10 years since they figured out how to: 1) Use convolutional layers, ReLUs, and a few other tricks to ameliorate the vanishing gradient problem, 2) Perform continuous, high-dimensional stochastic gradient descent on graphics cards, and 3) Apply these things via stochastic grad-student descent to sufficiently massive datasets that even the most brute-force models and training methods (back…
Convolutional layers have been around longer than that and don't have much to do with the vanishing gradient problem, which was never that big a problem to begin with. ReLUs are helpful, but they do not lead to THAT big of a speedup. Convolutional layers have definitely had a big impact for accuracy in image problems and sound problems. I think #2 is very important; they speed up both training and evaluation by a factor of 100 compared to a modern CPU.
None of these developments really explain why AI has become so popular though. Neural nets don't actually perform very well for most problems (image and speech problems being the major exceptions) compared to tree-based methods. I think it has more to do with the development of easy-to-use machine learning libraries as well as distributed processing libraries and hardware.
Re: Artificial Intelligence Software Is Booming, But Why Now?
#8Because it's been just about 10 years since they figured out how to: 1) Use convolutional layers, ReLUs, and a few other tricks to ameliorate the vanishing gradient problem, 2) Perform continuous, high-dimensional stochastic gradient descent on graphics cards, and 3) Apply these things via stochastic grad-student descent to sufficiently massive datasets that even the most brute-force models and training methods (back…
I've been thinking a lot about this lately, especially as I've been learning more about the difference between "nice" optimization problems with differentiable cost functions, and "messy" optimizations with less ideal cost functions.
Is there any reading material you can recommend on the recent history of ML/AI for someone like me, with a dilettante's understanding of these problems and an interest in optimization?
Re: Artificial Intelligence Software Is Booming, But Why Now?
#9TLDR from article which supplement's eli_gottlieb's comment:
"Much of today’s A.I. boom goes back to 2006, when Amazon started selling cheap computing over the internet. Those measures built the public clouds of Amazon, Google, IBM and Microsoft, among others. That same year, Google and Yahoo released statistical methods for dealing with the unruly data of human behavior. In 2007, Apple released the first iPhone, a device that began a boom in unruly-data collection everywhere."
The combination of smartphone + cloud created a virtuous (for AI, that is) cycle of unending data collection, which fed the improvements in theoretical research models which needed data at a larger scale to be validated.
Re: Artificial Intelligence Software Is Booming, But Why Now?
#10TLDR from article which supplement's eli_gottlieb's comment: "Much of today’s A.I. boom goes back to 2006, when Amazon started selling cheap computing over the internet. Those measures built the public clouds of Amazon, Google, IBM and Microsoft, among others. That same year, Google and Yahoo released statistical methods for dealing with the unruly data of human behavior. In 2007, Apple released the first iPhone, a d…
I remember doing AI work in 2005-2007 (genetic programming and similar things) and one of the hardest thing back then was that good quality training data sets were really hard to come by. I can second the idea that data makes a huge difference, and there's tons of it today.