Researchers: Are we on the cusp of an ‘AI winter’?
1–10 of 123 posts
Re: Researchers: Are we on the cusp of an ‘AI winter’?
#2I think the total economic impact of AI will be greatest for tasks that output high-dimensional data, such as GANs. For the simple reason that it can replace a lot more human labor. A great many jobs could be augmented with such tech.
Furthermore, I think the results from GPT-2 and similar language models show that researchers have found a scalable technique for sequence understanding. They are likely to just work better and better as you throw more data and training time at them. Imagine what GPT-2 could do if trained on 1000x more data and had 1000x more parameters. It would probably show deep understanding in a great variety of ideas and if prompted properly would probably pass a lot of Turing tests. There is evidence that this type of model learns somewhat generally, that is, structures it learns in one domain do help it learn faster in other domains. I am not sure exactly what would be possible with such a model, but I suspect it would be extremely impressive and meaningful economically.
I think we are likely to see that type of progress in the next year or two, and for there to be no AI winter.
Re: Researchers: Are we on the cusp of an ‘AI winter’?
#3No.
Maybe an "AI Fall", but I doubt there will ever be another true "AI Winter". The AI we have today is too good, and creates too much value... at this point, there is no longer any question as to whether or not there is value in continuing to research and invest in AI.
What will happen, almost without doubt, is that particular niches within the overall rubric of "AI" will go in and out of vogue, and investment in particular segments will fluctuate. For example, the steam will run out of the "deep learning revolution" at some point, as people realize that DL alone is not enough to make the leap to systems that employ common sense reasoning, have a grasp of intuitive physics, have an intuitive metaphysics, and have other such attributes that will be needed to come close to approximating human intelligence.
Disclaimer: credit for the observation about "intuitive physics" and "intuitive metaphysics" goes to Melanie Mitchell, via her recent AI Podcast interview with Lex Fridman.
One other observation... while we still don't know how far away AGI is (much less ASI), or even if it's possible, the important thing is that we don't need AGI to do many amazing and valuable things. I also doubt many people are actually all that disillusioned that we aren't yet living in The Matrix (or are we???).
Re: Researchers: Are we on the cusp of an ‘AI winter’?
#4I'm not sure about the pace of progress in research, but as an ML engineer at a startup who has been following developments, even if AI research stalls out completely, we've been given a huge set of amazing tools to apply to all kinds of technical problems for years to come.
I also think that even in the absence of massive breakthroughs, there's still plenty of work to be done during a "winter" in filling in the gaps of understanding between various SOTA advances. I think it may be the nature of science to advance in a sort of unbalanced tree of breakthroughs, where we drill down on certain popular and lucrative branches for a while before coming back to fill out and balance the width of the tree, if that analogy makes sense.
Just between transformers/autoencoders, GANs, classic classifiers, and combinations thereof I think we are already poised to see neural networks change society in the next ten or so years in a way similar to the influence of the internet. Especially if hardware and cloud computing continues to scale.
Re: Researchers: Are we on the cusp of an ‘AI winter’?
#5I think there's an interesting disconnect right now between research and practice. Cutting-edge research does feel like it's reaching a plateau - across most AI fields even "major" breakthroughs are only gaining a couple percentage points and we're probably starting to hit the limits of what current approaches can achieve. When the state-of-the-art is 97% on a task, there's only so much room for improvement. Yoav Goldberg posted a tweet about Facebook's RoBERTa model that summed it up pretty well: "oh wow seems like this boring public hyperparameter search is going to take a while" [1]. There's a vague feeling of "What's next?" now that all the benchmarks are fairly well-solved but AI in general clearly doesn't feel solved.
However, state-of-the-art models aren't really used in production yet. I think the trend of "use AI/ML to solve X" has only started to pick up in the past 2 years, and it'll continue well into the 2020s. The process of taking research models and putting them into production is not standardized yet, and many models don't even really work in production - if your model takes a second to do an inference step that's fine for research but maybe not for a real product.
I think in the next decade, on the research side, benchmarks will be beaten less often, and instead there will be more focus on trying out radically new things, understanding weaknesses in current techniques, and finding new measurements that assess those weaknesses. On the industry side, there will still be lots of cool and exciting new achievements as already-known techniques are applied to old problems that haven't been addressed by AI yet.
As an aside, this was the first time in my life that I read the phrase "10s" referring to the 2010-2019. Kind of an odd-feeling moment!
Re: Researchers: Are we on the cusp of an ‘AI winter’?
#6Research progress might slow down for a bit in some areas of machine learning but the commercialization of existing technology will keep us busy for the next 10 years.
Unlike the past two winters, deep learning is actually enabling a ton of applications that wouldn't have been possible otherwise and we now live in a world with a lot more data and computers to apply it to.
Re: Researchers: Are we on the cusp of an ‘AI winter’?
#7I'm not sure about the pace of progress in research, but as an ML engineer at a startup who has been following developments, even if AI research stalls out completely, we've been given a huge set of amazing tools to apply to all kinds of technical problems for years to come. I also think that even in the absence of massive breakthroughs, there's still plenty of work to be done during a "winter" in filling in the gaps…
Yeah, on the applied side there's still a ton of work left to be done to make productionization of machine learning systems easier. We still lack good tooling for data annotation, experiment tracking, model calibration/evaluation and monitoring.
Re: Researchers: Are we on the cusp of an ‘AI winter’?
#8I've been waiting for the hype and marketing to collapse for a few years.
Probably the fastest way to tarnish public perception of AI would be to keep pushing "AI-enhanced" products in front of the consumer as has been done. These things tend to demo well and have a nice cool factor for the first fifteen minutes or so, but after any kind of prolonged usage the limitations and rough-edges come up quick.
Re: Researchers: Are we on the cusp of an ‘AI winter’?
#9I think the total economic impact of AI will be greatest for tasks that output high-dimensional data, such as GANs. For the simple reason that it can replace a lot more human labor. A great many jobs could be augmented with such tech. Furthermore, I think the results from GPT-2 and similar language models show that researchers have found a scalable technique for sequence understanding. They are likely to just work be…
While I don't think there's going to be an AI winter either, I don't think GPT-2 will achieve sentience or anything close to it.
And that's for the same reason that no matter how much data they feed Tesla's self-driving AI, it will still try to kill you now and then. The problem space is just too big. All the people I know in this space don't think it will be solved for at least a decade and maybe not even then.
But I do suspect the 2020s will see the creation of agents combining classical algorithms with deep neural networks to do amazing things in domains that are closed and constant. But they're all going to be glorified (yet wonderful) unitaskers.
The only thing that worries me is that I don't trust FAANG to do the right thing ever anymore, and it's amazing to me that so many have opted into the panopticon of things in exchange for the ability to order stuff and turn their gadgets on and off.
Re: Researchers: Are we on the cusp of an ‘AI winter’?
#10I think the total economic impact of AI will be greatest for tasks that output high-dimensional data, such as GANs. For the simple reason that it can replace a lot more human labor. A great many jobs could be augmented with such tech. Furthermore, I think the results from GPT-2 and similar language models show that researchers have found a scalable technique for sequence understanding. They are likely to just work be…
Does GPT-2 really "understand" anything? I feel like this is pretty quickly going to devolve into a semantic argument, but having interacted with some trained GPT-2 models, it seems to produce only what Orwell would have called duckspeak[0]. There's very clearly no mind behind the words, so it's hard for me to credit it with understanding.