Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning: The Great Stagnation

marksaroufim.substack.com

Post reply on HN
Print view View original

71–80 of 227 posts

Re: Machine Learning: The Great Stagnation

#71

Post by RGamma »

Rochus wrote: post #28
Tenoke wrote: post #21

Earlier quoted context omitted.

Gary Marcus has been proven wrong on claims he's made and is mostly just nay-saying with adhoc reasoning made up to support it. >appears less biased than others who benefit from the current high level of investment in DNN. Yes, instead he blatantly tries to benefit from the counter-investment in AI skepticism.

There somehow seems to be an unwritten law of the Python generation like "Thou shalt not criticise Machine Learning". Or is there a better explanation for the emotions that flare up every time someone dampens the exaggerated expectations and reminds us of earlier research in the field of linguistics or AI?

Not your problem. Just let 'em be stuck in their local optimum (probably better this way) :P

Re: Machine Learning: The Great Stagnation

#72

Post by WanderPanda »

mlthoughts2018 wrote: post #26
WanderPanda wrote: post #11

Earlier quoted context omitted.

Just a quick belief of myself (might change tomorrow): ML winter will not come until we hit the flattening of the specialized hardware s-curve. I know people believe that ML can still scale in funding (for even bigger models than GPT-3 with current hardware) but that shouldn’t be our hope. I also can not imagine that the exponential efficiency increase in architectures with optimized structure can continue for a coup…

Most commercial use of ML, including neural nets, is small data application-specific business logic. Think phishing / spam / fraud detection, anomaly detection, semantic search, image search, keyword labeling, classifying or segmenting customer data. These applications have well-understood business value propositions. Much, much less often the ML application focused on truly large data. I think we might see a winter…

I agree, or at least I hope you are right. I have some doubts about investment into ML without the hype (at least for legacy big corps) but probably their hype-driven ML efforts are/were misguided by poor incentives. But I agree again that companies that invest in ML from a grassroots/first principles way should run circles around the legacy ones. I think the „boring“ part you mention might be what I refer to as ML/AI winter. But then again do you think there is much space left in the ML toolbox apart from neural networks? I think at least for e.g. computer vision we can agree that neural nets ate all the cake, right?

Re: Machine Learning: The Great Stagnation

#73

Post by lalo2302 »

gbolcer wrote: post #69

Graduate Student Descent -- there's also the other saying: Every moment you spend writing, publishing, or working on your dissertation is a moment that you are getting behind on doing unique and cutting edge research.

The only difference between doing science and fooling around is when you write it down

Re: Machine Learning: The Great Stagnation

#74

Post by bidirectional »

otabdeveloper4 wrote: post #19
superbcarrot wrote: post #12

Earlier quoted context omitted.

That's unlikely. There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. The worst case for the field is that progress slows down, people realise that their expectations were unrealistic and the hype inevitably dies down. Which has to happen eventually. So even if ML isn't the hottest thing or a massively growing field, it will still be us…

> There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. If you could name three of them I'd be really grateful. Serious question; everything surrounding ML seems to be only good for (non-monetizable) art projects. As art it is amazing, not going to lie, but "commercial use" seems like a huge stretch.

As far as I'm aware, ML is driving a pretty sizeable amount of new functionality at the big tech companies. How do you think Google does search and translate, Netflix recommends films, Tesla drives your car, Siri recognises your voice, etc. etc.

I see the sentiment you've expressed a lot, and feel it speaks to a massive disconnect amongst developers. Most people are interacting with ML systems dozens if not hundreds of times a day.

Re: Machine Learning: The Great Stagnation

#75

Post by bendergarcia »

funcDropShadow wrote: post #38
Ericson2314 wrote: post #25

I still feel like much of AI is a plot to dumb down the modern economy. We want our business people to be just as effective as our quants; we want nothing to require real intellectual labor. The idea that you traditionally have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and all the more so because you are utterly dependent…

> Now, that's not to say I am against labor saving devices. That's good to know. Because "labor saving devices" is by no means the field of machine learning. A hammer is a "labor saving device" if all you have are rocks. I've got the impression that a lot of people conflate machine learning and robotics with labor saving in general. Of course compared to the state of art machine learning let's us hope to find magical…

Agreed I find that ml is a meta tool. You gotta Know the tool you need, then use ml to build it. Build the wrong tool and there’s little value. The right tool will make people more productive.

Re: Machine Learning: The Great Stagnation

#76

Post by gnramires »

knuthsat wrote: post #9

There's a nice talk by Yann LeCun where he goes on to explain really well why deep learning has such fast progress. [0] He goes on to explain how theory always comes later. I thought that information theory came before practice but turns out it also came after. (there was a bunch of heuristics for sending messages with teletypes) A nice example is Roman technological advances in architecture and materials that predat…

It's not so simple as "what comes first?" (theory or experiment). What generally happens is that we stumble upon something, build a rudimentary but useful or interesting piece, then try to understand it, generalize it, and expand it (and usually succeed, with science). Both happen iteratively.

Information theory came after the telegraph and early communication systems. However, we could not have built modern communication devices without Information theory (the insights and design principles). We build, then we theorize, then we build better, etc. it's not a simple procedure. Computers were developed similarly: there were all sorts of ad-hoc logical apparatus, we built boolean theory to explain it, and then we did all sorts of experiments trying to build computers. Their architects were largely mathematicians with very good ideas of mathematical design principles (and creative new mathematical ideas), not a group stringing together electrical elements and seeing what happens. The same goes for the development of ML/Deep learning, and many other technological marvels.

Although "accidental" discoveries do happen, they happen from a methodical set, with knowledge, intuition, and good priors.

Re: Machine Learning: The Great Stagnation

#78

Post by EVa5I7bHFq9mnYK »

Ericson2314 wrote: post #25

I still feel like much of AI is a plot to dumb down the modern economy. We want our business people to be just as effective as our quants; we want nothing to require real intellectual labor. The idea that you traditionally have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and all the more so because you are utterly dependent…

That reminds me how SQL was advertised - "Now you can program computers in plain English and fire all your programmers!". Fast forward, and today good luck getting a job as a Microsoft SQL programmer if you only know Oracle SQL.

Re: Machine Learning: The Great Stagnation

#79

Post by Ericson2314 »

wpietri wrote: post #70
Ericson2314 wrote: post #25

I still feel like much of AI is a plot to dumb down the modern economy. We want our business people to be just as effective as our quants; we want nothing to require real intellectual labor. The idea that you traditionally have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and all the more so because you are utterly dependent…

This seems fair to me. The executive view of ML is "can you do me a magic?" And as this article's "Graduate Student Descent" bit makes clear, the worker response is often to semi-randomly perturb code, show some graphs, and say, "Is this a magic?" For me, most software development is about finding something boring and laborious. We get a computer to do the work so humans can level up and work on something requiring a…

> Especially people in the managerial caste, as the reigning dogma there is that management is a universal skill. Details are for the little people.

Exactly. Machine learning is the perfect ideological duel. It's "universal labor" for "universal management", and both sides are equally illiterate in the ways of the world.

Re: Machine Learning: The Great Stagnation

#80

Post by AndrewKemendo »

b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

>We've been in an exciting deep learning craze for a while, but it's silly to expect it to last forever. Back to the grind now.

This was effectively my response to hardmaru when this topic came up on reddit [1]

Basically 2010-2018 was an open field for ML/DL research with old(ish) methods being rapidly applied to low hanging fruit and large datasets with newly cheap compute.

Deepmind and others are actually making new methods but by and large are remixes of those same old approaches.

The majority of different research out there trying other approaches (Numenta, OpenCog, Causal Calculus, anything Schmidhuber etc...) don't really get any love because it doesn't fit within the mass tensorflow/torch framework.

[1]https://twitter.com/AndrewKemendo/status/1349387455552745473

Post reply on HN