Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning: The Great Stagnation

marksaroufim.substack.com

Post reply on HN
Print view View original

121–130 of 227 posts

Re: Machine Learning: The Great Stagnation

#121

Post by screye »

mlthoughts2018 wrote: post #50
screye wrote: post #32

There is about to be a 'great pivot' in ML. There has been a rabid frenzy of throwing money at anything that has ML in it. Soon investors and CEOs will realize that ML is effective in narrow ways and that not everything needs ML. They will also realize that 1 ML team + ML as a service (Azure ML, Sagemaker, Google AI platform) is cheaper and works more reliably. The services will keep improving and an underpaid medioc…

I am an experienced ML manager in a large ecommerce company, and I mostly agree with you, and I can’t wait for this to happen - and I think people just entering college or grad school for ML should not fear it. It’s a good thing. Right now, there is so much misunderstanding about what ML is, what resources it needs, and how it works that the corporate environment is very stressful. ML jobs are well paid, but they are…

> ML jobs are well paid, but they are NOT fun. No one understands ML devops & the infra needs to enable tight experimentation loops. Existing observability and telemetry systems are wildly bad for model training, reproducibility or any form of online or semi-online learning. As an ML engineer you’ll have to take on huge workloads of devops, infra, tooling, data munging. I’ve seen more than a few brilliant ML engineers burnout and quit because of this.

Can I cry? I feel so understood right now.

I love my job in ML, the subject matter is fun, but there is so such a huge burden of expectations on a team's titular data scientist. It is exciting in a 'mid 90s during the web revolution' sort of wild-west way, but you also have the cynicism of the mature Software field. A good ML engineer is worth their weight in gold.

________

I also wrote this in a pseudo-fictional dystopian sense. A 'If I was an ML pessimist' take on the the state of things.

The other comments made to the parent I originally posted, are great counter arguments. (2012-14: Alexnet, 14-16: Deep LSTMs, 16-18: Resnet,M-RCNN,Yolo 18-20: Tranformers, 2020+: Alphafold,GPT3,CLIP, et al.) Deep learning has been improving pretty linearly over the last decade. If I was looking at it in a naively statistical sense, then ML will actually be able to match the rising supply of ML scientists with a rising demand. That's the optimistic take though. In that case it will actually feel like being a programmer in the 90s, in that a couple pivots can propel you to multi millionaire.

Re: Machine Learning: The Great Stagnation

#123

Post by screye »

ramoz wrote: post #61
screye wrote: post #32

There is about to be a 'great pivot' in ML. There has been a rabid frenzy of throwing money at anything that has ML in it. Soon investors and CEOs will realize that ML is effective in narrow ways and that not everything needs ML. They will also realize that 1 ML team + ML as a service (Azure ML, Sagemaker, Google AI platform) is cheaper and works more reliably. The services will keep improving and an underpaid medioc…

Mostly disagree. > CEOs will realize that ML is effective in narrow ways and that not everything needs ML. Any stable business isn't unjustifiably syncing costs here. I project FY21 rise an AI-funded efforts in large businesses. > They will also realize that 1 ML team + ML as a service Yes/no. This is has more platform implications vs actual ML. > ML teams might lose jobs Assumes ML Jobs only do some form of R&D. Dat…

The optimistic side of me agrees with you.

For my career I say, "tere u me shakkar". ("let there be sugar in your mouth", ie. let your words come true)

Re: Machine Learning: The Great Stagnation

#124

Post by JPKab »

b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

Just to be clear, there's a lot of researchers making money that aren't producing squat. There are many AI ethicists I admire, and there are many that can't code, can't produce, and just spend their time getting into arguments with people like Yann LeCunn, who actually have produced groundbreaking research.

Re: Machine Learning: The Great Stagnation

#125

Post by b3kart »

tracyhenry wrote: post #107
b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

> It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). Is the deep learning really the result of incremental research? The SOTA chasing frenzy comes after the discovery of deep learning. The motivation of incremental research can hardly be justified as to discover the next deep learning, although they might do.

Of course it is, if we agree that incremental research isn't necessarily "SOTA chasing". Deep learning, and in fact all scientific breakthroughs, did not materialize out of nowhere. They're a culmination of a long sequence of scientific papers (some more incremental than others), conference presentations, poster sessions, hallway conversations, even coffee chats.

It's not some dude disappearing into the forest for a few years and coming back with a revolutionary idea (maybe in movies). Everything that helped shape the scientist's thinking has contributed to the idea, even if sometimes the idea is so revolutionary that it's hard to see the direct link. Besides, show me a scientific paper with no references. :-)

It's a shame that this is rarely acknowledged and more often than not we talk about "that guy who invented X and got a Nobel prize for it". But us mere mortals can take solace in knowing that even if we didn't change the world with our own ideas, it's possible that we've influenced someone who has.

Re: Machine Learning: The Great Stagnation

#126

Post by benlivengood »

otabdeveloper4 wrote: post #19
superbcarrot wrote: post #12

Earlier quoted context omitted.

That's unlikely. There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. The worst case for the field is that progress slows down, people realise that their expectations were unrealistic and the hype inevitably dies down. Which has to happen eventually. So even if ML isn't the hottest thing or a massively growing field, it will still be us…

> There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. If you could name three of them I'd be really grateful. Serious question; everything surrounding ML seems to be only good for (non-monetizable) art projects. As art it is amazing, not going to lie, but "commercial use" seems like a huge stretch.

Speech-to-text. Every voice assistant, many phone trees, assistive software.

Facial recognition. iphone face unlock, photo tagging, etc.

Behavior prediction for advertising. Ad quality scores basically.

Re: Machine Learning: The Great Stagnation

#127

Post by divtiwari »

mlthoughts2018 wrote: post #50
screye wrote: post #32

There is about to be a 'great pivot' in ML. There has been a rabid frenzy of throwing money at anything that has ML in it. Soon investors and CEOs will realize that ML is effective in narrow ways and that not everything needs ML. They will also realize that 1 ML team + ML as a service (Azure ML, Sagemaker, Google AI platform) is cheaper and works more reliably. The services will keep improving and an underpaid medioc…

I am an experienced ML manager in a large ecommerce company, and I mostly agree with you, and I can’t wait for this to happen - and I think people just entering college or grad school for ML should not fear it. It’s a good thing. Right now, there is so much misunderstanding about what ML is, what resources it needs, and how it works that the corporate environment is very stressful. ML jobs are well paid, but they are…

Bravo Sir!!! You have put it brilliantly!!

Re: Machine Learning: The Great Stagnation

#128

Post by officehero »

screye wrote: post #32

There is about to be a 'great pivot' in ML. There has been a rabid frenzy of throwing money at anything that has ML in it. Soon investors and CEOs will realize that ML is effective in narrow ways and that not everything needs ML. They will also realize that 1 ML team + ML as a service (Azure ML, Sagemaker, Google AI platform) is cheaper and works more reliably. The services will keep improving and an underpaid medioc…

> become boring

80-95% of "ML" has always been "boring" and any data scientist/ML-engineer worth their salt know this. This also concerns what you refer to as "pure ML job". It only takes fresh grads and juniors a couple of projects to realize the true meaning of "data cleaning", "outlier detection" "robustness" and their likes - it's painstaking work.

Re: Machine Learning: The Great Stagnation

#129

Post by ChrisRackauckas »

JHonaker wrote: post #110

I kind of agree with the author's major sentiment: that ML research is stuck in a rut with incremental improvement. However, the longer the article goes on, the less and less I agree with any of their statements. They start of criticizing the incremental improvers. They advocate later that if "stack more layers" beats a method, the method isn't good while completely ignoring anything other than the standard SOTA metr…

>some ODE solvers

Some? Which ones aren't? Could you open an issue?

Re: Machine Learning: The Great Stagnation

#130

Post by kkylin »

b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

I think you are exactly right, and want to add a couple things ():

- It's easy to recognize scientific / technological revolutions in hindsight, but at the time they're anything but. I think most recognize the importance of persistence in the progenitor of an idea. What's often missed in these discussions is how important the subsequent incremental progress is: working out consequences of a theoretical insight, figuring out what you can build on top of a new tech, etc.

- I don't have a strong opinion about the optimal amount of "risk," but I do think making risk existential (as in one would starve without a research breakthrough) would have the opposite effect, because while we all say we like people to take risks, what we really mean is like people to take risks and suceeed. And yes, too little risk can breed complacency.

And yes, convolutions are a specific kind of linear transformation, whereas matrices represent linear transformations on vector spaces. The specific structure is that convolutions represent linear transformations that are translation-invariant, a property that many types of data (e.g., images) have, at least approximately.

() My perspective as an academic but not a computer scientist.

Post reply on HN