Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning: The Great Stagnation

marksaroufim.substack.com

Post reply on HN
Print view View original

11–20 of 227 posts

Re: Machine Learning: The Great Stagnation

#11

Post by WanderPanda »

marvel_boy wrote: post #2

Winter is coming, again.

Just a quick belief of myself (might change tomorrow): ML winter will not come until we hit the flattening of the specialized hardware s-curve. I know people believe that ML can still scale in funding (for even bigger models than GPT-3 with current hardware) but that shouldn’t be our hope. I also can not imagine that the exponential efficiency increase in architectures with optimized structure can continue for a couple more years (happy to be proven wrong here). Also the scaling in dataset sizes will ultimately halt (or might be already) for supervised learning which might give a bump for reinforcement learning where there is usually unlimited data (due to simulation)

Re: Machine Learning: The Great Stagnation

#12

Post by superbcarrot »

marvel_boy wrote: post #2

Winter is coming, again.

That's unlikely. There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. The worst case for the field is that progress slows down, people realise that their expectations were unrealistic and the hype inevitably dies down. Which has to happen eventually. So even if ML isn't the hottest thing or a massively growing field, it will still be used for a long time.

Re: Machine Learning: The Great Stagnation

#13

Post by Rochus »

laichzeit0 wrote: post #3

> It’s important to avoid becoming Gary Marcus and criticize existing technique that work without proposing something else that works even better. Hilarious. Has Gary Marcus actually done anything, in practical terms, like actual code or something, that outperforms the DL approaches he attacks so viciously?

Marcus argues convincingly and appears less biased than others who benefit from the current high level of investment in DNN.

It seems to be true that no better proposals for solutions have come from his side so far. But I think his criticism per se is valuable, especially his reminder that one cannot simply ignore sixty years of research.

EDIT: and of course he is not the only renowned scientist who calls for reflection; here is a quote from an interview with Judea Pearl: "AI is currently split. First, there are those who are intoxicated by the success of machine learning and deep learning and neural nets. They don’t understand what I’m talking about. They want to continue to fit curves. But when you talk to people who have done any work in AI outside statistical learning, they get it immediately. I have read several papers written in the past two months about the limitations of machine learning." (https://www.theatlantic.com/technology/archive/2018/05/machi...)

Re: Machine Learning: The Great Stagnation

#14

Post by rowanG077 »

jarym wrote: post #4

> Academics sacrifice material opportunity costs in exchange for intellectual freedom. Most academics I’ve come across only think they’re doing this. My perception is they are too insecure about their self-worth to pursue material opportunities. I admit, the number of academic types I know is not vast so maybe it’s too small a subset to make any judgments

The core problem is that you don't have intellectual freedom. You won't get funded if you are not researching the hot new thing.

Re: Machine Learning: The Great Stagnation

#15

Post by osipov »

> If an idea is guaranteed to work then it moves from the realm of research to engineering.

The idea of the Dyson Sphere is guaranteed to work, so I guess the author is under the impression that there will not be research involved.

Re: Machine Learning: The Great Stagnation

#17

Post by dimitrios1 »

rowanG077 wrote: post #14
jarym wrote: post #4

> Academics sacrifice material opportunity costs in exchange for intellectual freedom. Most academics I’ve come across only think they’re doing this. My perception is they are too insecure about their self-worth to pursue material opportunities. I admit, the number of academic types I know is not vast so maybe it’s too small a subset to make any judgments

The core problem is that you don't have intellectual freedom. You won't get funded if you are not researching the hot new thing.

Exactly, and that's assuming you are one of the very select few that gets to even select and lead the research areas. Most of the opportunities are working along side professors and department heads that already have an idea of what you should be working on. If you are at one of the elite universities with billion dollar endowments that don't have a lot of these funding problems, then you are working on what the corporate donors suggest the University work on. The university will sell it as "staying relevant for the future job market"

Re: Machine Learning: The Great Stagnation

#19

Post by otabdeveloper4 »

superbcarrot wrote: post #12
marvel_boy wrote: post #2

Winter is coming, again.

That's unlikely. There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. The worst case for the field is that progress slows down, people realise that their expectations were unrealistic and the hype inevitably dies down. Which has to happen eventually. So even if ML isn't the hottest thing or a massively growing field, it will still be us…

> There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere.

If you could name three of them I'd be really grateful. Serious question; everything surrounding ML seems to be only good for (non-monetizable) art projects.

As art it is amazing, not going to lie, but "commercial use" seems like a huge stretch.

Post reply on HN