Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning: The Great Stagnation

marksaroufim.substack.com

Post reply on HN
Print view View original

21–30 of 227 posts

Re: Machine Learning: The Great Stagnation

#21

Post by Tenoke »

Rochus wrote: post #13
laichzeit0 wrote: post #3

> It’s important to avoid becoming Gary Marcus and criticize existing technique that work without proposing something else that works even better. Hilarious. Has Gary Marcus actually done anything, in practical terms, like actual code or something, that outperforms the DL approaches he attacks so viciously?

Marcus argues convincingly and appears less biased than others who benefit from the current high level of investment in DNN. It seems to be true that no better proposals for solutions have come from his side so far. But I think his criticism per se is valuable, especially his reminder that one cannot simply ignore sixty years of research. EDIT: and of course he is not the only renowned scientist who calls for reflect…

Gary Marcus has been proven wrong on claims he's made and is mostly just nay-saying with adhoc reasoning made up to support it.

>appears less biased than others who benefit from the current high level of investment in DNN.

Yes, instead he blatantly tries to benefit from the counter-investment in AI skepticism.

Re: Machine Learning: The Great Stagnation

#22

Post by ipunchghosts »

jarym wrote: post #4

> Academics sacrifice material opportunity costs in exchange for intellectual freedom. Most academics I’ve come across only think they’re doing this. My perception is they are too insecure about their self-worth to pursue material opportunities. I admit, the number of academic types I know is not vast so maybe it’s too small a subset to make any judgments

I went a very uncommon route in my career: I started phd school after 14 years of industry. My motivation was I was tired of creating new and interesting things in industry only to have them be killed by politics. So, my idea never saw the light of day san a few dozen people. My thought was that if i have a phd, I am rewarded in my career for publishing these thoughts. If they are published, maybe some organization smarter than mine would use the ideas. I am a true scientist at heart in that i want to know where my ideas break and a good way to know that is to have a lot of public scrutiny over them.

Re: Machine Learning: The Great Stagnation

#24

Post by [deleted] »

rowanG077 wrote: post #14
jarym wrote: post #4

> Academics sacrifice material opportunity costs in exchange for intellectual freedom. Most academics I’ve come across only think they’re doing this. My perception is they are too insecure about their self-worth to pursue material opportunities. I admit, the number of academic types I know is not vast so maybe it’s too small a subset to make any judgments

The core problem is that you don't have intellectual freedom. You won't get funded if you are not researching the hot new thing.

[deleted]

Re: Machine Learning: The Great Stagnation

#25

Post by Ericson2314 »

I still feel like much of AI is a plot to dumb down the modern economy. We want our business people to be just as effective as our quants; we want nothing to require real intellectual labor.

The idea that you traditionally have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and all the more so because you are utterly dependent on them. ML breaks their control over the means of production.

Now, that's not to say I am against labor saving devices. I most certainly am for them, but an an economy in which everyone is in a deep learning arms race is an irrational shit show that could only result in less productivity.

(It's possible a single central planner AI could do better, because at least the training data would be "real world" and not output of other deep learning black box actors. But of course single-planner economies have a huge amount of other downsides.)

Re: Machine Learning: The Great Stagnation

#26

Post by mlthoughts2018 »

WanderPanda wrote: post #11
marvel_boy wrote: post #2

Winter is coming, again.

Just a quick belief of myself (might change tomorrow): ML winter will not come until we hit the flattening of the specialized hardware s-curve. I know people believe that ML can still scale in funding (for even bigger models than GPT-3 with current hardware) but that shouldn’t be our hope. I also can not imagine that the exponential efficiency increase in architectures with optimized structure can continue for a coup…

Most commercial use of ML, including neural nets, is small data application-specific business logic. Think phishing / spam / fraud detection, anomaly detection, semantic search, image search, keyword labeling, classifying or segmenting customer data. These applications have well-understood business value propositions. Much, much less often the ML application focused on truly large data.

I think we might see a winter in super big applications like self-driving cars or voice assistants, but ML in general is just a boring, non-controversial business tool with hundreds of valuable applications.

You’ll still need statistical specialists to train and operate models and ensure systems avoid pitfalls like overfitting, poor convergence, multicollinearity, confounders, etc.

So I doubt this will have much impact on ML job market. Companies that invest in ML will continue to run circles around companies that don’t. You’ll just see the unjustified over-focus on SOTA neural networks die down and become just another boring tool in the toolbox like everything else in ML.

Re: Machine Learning: The Great Stagnation

#27

Post by cscurmudgeon »

laichzeit0 wrote: post #3

> It’s important to avoid becoming Gary Marcus and criticize existing technique that work without proposing something else that works even better. Hilarious. Has Gary Marcus actually done anything, in practical terms, like actual code or something, that outperforms the DL approaches he attacks so viciously?

I am a DL researcher and practitioner.

Any response to criticism that brushes off the criticism based on the source of the criticism is bad and not in the scientific spirit. Progress comes by asking questions and pointing out flaws. You don't need to have an answer to the question before you ask the question.

Marcus's criticisms are valid. Most DL researchers have a huge blind spot. It is not good for the field.

Re: Machine Learning: The Great Stagnation

#28

Post by Rochus »

Tenoke wrote: post #21
Rochus wrote: post #13

Earlier quoted context omitted.

Marcus argues convincingly and appears less biased than others who benefit from the current high level of investment in DNN. It seems to be true that no better proposals for solutions have come from his side so far. But I think his criticism per se is valuable, especially his reminder that one cannot simply ignore sixty years of research. EDIT: and of course he is not the only renowned scientist who calls for reflect…

Gary Marcus has been proven wrong on claims he's made and is mostly just nay-saying with adhoc reasoning made up to support it. >appears less biased than others who benefit from the current high level of investment in DNN. Yes, instead he blatantly tries to benefit from the counter-investment in AI skepticism.

There somehow seems to be an unwritten law of the Python generation like "Thou shalt not criticise Machine Learning". Or is there a better explanation for the emotions that flare up every time someone dampens the exaggerated expectations and reminds us of earlier research in the field of linguistics or AI?

Re: Machine Learning: The Great Stagnation

#29

Post by Tenoke »

otabdeveloper4 wrote: post #19
superbcarrot wrote: post #12

Earlier quoted context omitted.

That's unlikely. There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. The worst case for the field is that progress slows down, people realise that their expectations were unrealistic and the hype inevitably dies down. Which has to happen eventually. So even if ML isn't the hottest thing or a massively growing field, it will still be us…

> There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. If you could name three of them I'd be really grateful. Serious question; everything surrounding ML seems to be only good for (non-monetizable) art projects. As art it is amazing, not going to lie, but "commercial use" seems like a huge stretch.

Here's 3 off the top of my head, but there's more especially when you get into less flashy territory.

Translation (Google translate, DeepL)

Automatically generated product descriptions, sometimes also edited by humans (Alibaba)

Image Tagging (Facebook photos)

Re: Machine Learning: The Great Stagnation

#30

Post by 314 »

marvel_boy wrote: post #2

Winter is coming, again.

Winter is always coming, the important question is when it will arrive. Current ML research is still destroying new problems with ease so the current velocity is high. It will slow down first, before people start to question why the new crop of problems are too hard, and then the cycle will start again.
Post reply on HN