Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning: The Great Stagnation

marksaroufim.substack.com

Post reply on HN
Print view View original

131–140 of 227 posts

Re: Machine Learning: The Great Stagnation

#131

Post by formalsystem »

Hey everyone this is OP, was a really nice surprise seeing my article generating so much discussion. I unfortunately don't think I'll have time to answer everyone but feel free to reply to this comment if you'd like to ask me anything.

It seems like the article was a bit polarizing, some of the comments made me realize I made a few imprecise statements and I'll fix those. Other comments didn't approve of my tone and that's a bit harder to fix since I tried writing this article in the same way I usually speak. I've written a lot of more dry content for my robotics ebook and I was purposeful in trying something different for this article.

Some of the comments actually generalized my observations to management and software more generally and it's always nice to see people taking my ideas further than I thought they could go.

One major thing I'd like to point out is I don't think "ML is dumb" is the right conclusion, there are lots of incentives making it so more of ML is stagnating but this is certainly not true for the field at large. Interesting ideas need to involve a certain amount of risk. The latter half of my article showcases a few ML adjacent projects which I think are absolutely fascinating.

And if you're interested in reading more stuff by me my robotics and machine learning ebook is very easy to read http://robotoverlordmanual.com/ and will teach you all you need to start building robots at home

Re: Machine Learning: The Great Stagnation

#132

Post by SilkStocking »

JPKab wrote: post #124
b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

Just to be clear, there's a lot of researchers making money that aren't producing squat. There are many AI ethicists I admire, and there are many that can't code, can't produce, and just spend their time getting into arguments with people like Yann LeCunn, who actually have produced groundbreaking research.

LeCon's work wasn't groundbreaking; he was saved from oblivion only by Moore's Law. LeCun stood still doing the same thing for decades. But when wave after wave of faster hardware arrived he was finally declared successful.

So if you want to credit success to dumb luck, or to stubbornness, or insanity (as Einstein said: "doing the same thing over and over again and expecting different results." then LeCon is your man. I don't know of any quote where LeCun says something to the effect of: "Well, I knew the hardware would speed up and then my neural net software would work exactly as I designed it to and manifest AI. So I waited twenty years."

IOW It wasn't the LeCunn's surfboard(his software) that handed him success, it was the wave(advancing hardware speeds):

https://www.wired.com/2017/04/race-make-ai-chips-everything-...

Re: Machine Learning: The Great Stagnation

#133

Post by ramoz »

screye wrote: post #123
ramoz wrote: post #61

Earlier quoted context omitted.

Mostly disagree. > CEOs will realize that ML is effective in narrow ways and that not everything needs ML. Any stable business isn't unjustifiably syncing costs here. I project FY21 rise an AI-funded efforts in large businesses. > They will also realize that 1 ML team + ML as a service Yes/no. This is has more platform implications vs actual ML. > ML teams might lose jobs Assumes ML Jobs only do some form of R&D. Dat…

The optimistic side of me agrees with you. For my career I say, "tere u me shakkar". ("let there be sugar in your mouth", ie. let your words come true)

Ha & to be fair I've been on and am on a call now trying to sugar coat this exact issue at scale with in-flux relatable scenarios.

Re: Machine Learning: The Great Stagnation

#134

Post by bnegreve »

dkislyuk wrote: post #108
b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

>> CNNs use convolutions which are a generalization of matrix multiplication. > A nitpick: CNNs are most definitely not a generalization of matrix multiplication. In fact, the opposite: you can view CNNs as a matrix multiplication with a particular matrix structure. Neither is really a generalization. Any matrix multiplication can be implemented with a convolution function, and yet any convolution can be represented…

Discrete convolutions are a special case of matrix multiplication in the sense that a discrete convolution can be formulated as the product between a Toeplitz matrix and the input signal.

[1] https://en.wikipedia.org/wiki/Toeplitz_matrix#Discrete_convo...

Re: Machine Learning: The Great Stagnation

#135

Post by [deleted] »

otabdeveloper4 wrote: post #119
bidirectional wrote: post #74

Earlier quoted context omitted.

As far as I'm aware, ML is driving a pretty sizeable amount of new functionality at the big tech companies. How do you think Google does search and translate, Netflix recommends films, Tesla drives your car, Siri recognises your voice, etc. etc. I see the sentiment you've expressed a lot, and feel it speaks to a massive disconnect amongst developers. Most people are interacting with ML systems dozens if not hundreds…

Question wasn't if ML is useful. The question is - does ML make money or create (financial) value for the company? The stuff you listed doesn't, it's just part of a moat for already established products that don't depend on ML for their market share. (It's not like Netflix will lose market share if they switch from ML to some other approach for their recommendation system.)

[deleted]

Re: Machine Learning: The Great Stagnation

#136

Post by p1esk »

AndrewKemendo wrote: post #80
b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

>We've been in an exciting deep learning craze for a while, but it's silly to expect it to last forever. Back to the grind now. This was effectively my response to hardmaru when this topic came up on reddit [1] Basically 2010-2018 was an open field for ML/DL research with old(ish) methods being rapidly applied to low hanging fruit and large datasets with newly cheap compute. Deepmind and others are actually making ne…

Those other approaches you mentioned don’t get much love because they don’t work. Their advocates worked on them for many years and have nothing to show for it.

Re: Machine Learning: The Great Stagnation

#137

Post by JHonaker »

ChrisRackauckas wrote: post #129
JHonaker wrote: post #110

I kind of agree with the author's major sentiment: that ML research is stuck in a rut with incremental improvement. However, the longer the article goes on, the less and less I agree with any of their statements. They start of criticizing the incremental improvers. They advocate later that if "stack more layers" beats a method, the method isn't good while completely ignoring anything other than the standard SOTA metr…

>some ODE solvers Some? Which ones aren't? Could you open an issue?

I was unsure if every ODE solver was able to be differentiated through. The "some" was me trying to avoid making a blanket statement about all, when I wasn't sure.

Sorry for the confusion.

Re: Machine Learning: The Great Stagnation

#138

Post by xpe »

rowanG077 wrote: post #14
jarym wrote: post #4

> Academics sacrifice material opportunity costs in exchange for intellectual freedom. Most academics I’ve come across only think they’re doing this. My perception is they are too insecure about their self-worth to pursue material opportunities. I admit, the number of academic types I know is not vast so maybe it’s too small a subset to make any judgments

The core problem is that you don't have intellectual freedom. You won't get funded if you are not researching the hot new thing.

This is clearly overstated. Some nuance would be appreciated.

Re: Machine Learning: The Great Stagnation

#139

Post by xpe »

jjk166 wrote: post #99
api wrote: post #56

Earlier quoted context omitted.

Trying to use ML to get rid of programmers will just replace them with ML experts who also have to be programmers to implement the models and munge all the data. These people will in turn have to be paid more than the original programmers were.

The goal of ML isn't so much to get rid of programmers as to get rid of specialized programmers. Right now if you want to solve a problem, you need someone with a deep understanding of that specific problem to develop a solution. For a complex problem like diagnosing cancer patients, you are talking about a team of people with decades if not centuries of combined experience in oncology on top of the actual programmin…

"The" goal of ML? No.

A goal? No. ML is a field, not something with agency.

One effect of ML is more generalization of prediction and inference problems. (I'm using inference in the statistical sense.)

Re: Machine Learning: The Great Stagnation

#140

Post by xpe »

Ericson2314 wrote: post #25

I still feel like much of AI is a plot to dumb down the modern economy. We want our business people to be just as effective as our quants; we want nothing to require real intellectual labor. The idea that you traditionally have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and all the more so because you are utterly dependent…

A plot? By whom?

ML, depending on the situation may refer to a field of research, a methodology, a set of tools, among other things. Only things with agency 'plot'.

Post reply on HN