Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning: The Great Stagnation

marksaroufim.substack.com

Post reply on HN
Print view View original

31–40 of 227 posts

Re: Machine Learning: The Great Stagnation

#31

Post by chriskanan »

dimitrios1 wrote: post #17
rowanG077 wrote: post #14

Earlier quoted context omitted.

The core problem is that you don't have intellectual freedom. You won't get funded if you are not researching the hot new thing.

Exactly, and that's assuming you are one of the very select few that gets to even select and lead the research areas. Most of the opportunities are working along side professors and department heads that already have an idea of what you should be working on. If you are at one of the elite universities with billion dollar endowments that don't have a lot of these funding problems, then you are working on what the corp…

If one has a supportive advisor and adequate non-grant support (e.g., a fellowship), it is possible to work on self-drive ambitious projects. However, typically an advisor only wants to invest significant time if the work is interesting to them. Also, often I've been approached by students who want to do their own thing in terms of high risk work, but they don't have adequate skills or enough existing work to graduate with. Mt goal is to ensure students advance knowledge, graduate, and get the kind of job they want. Ambitious projects that fail will significantly impair positive outcomes.

The billion dollar endowments don't usually go toward supporting research directly.

Re: Machine Learning: The Great Stagnation

#32

Post by screye »

There is about to be a 'great pivot' in ML.

There has been a rabid frenzy of throwing money at anything that has ML in it. Soon investors and CEOs will realize that ML is effective in narrow ways and that not everything needs ML.

They will also realize that 1 ML team + ML as a service (Azure ML, Sagemaker, Google AI platform) is cheaper and works more reliably. The services will keep improving and an underpaid mediocre ML-Engineering team can work to keep the production system up and running.

Basically, ML teams might lose jobs just as DB/Cluster admins did with the advent of serverless compute as service.

I expect it to (already happening) create a day trading company like hierarchy. The Fair/Brain/OpenAIs will pay 7 figures to the top grads to be first to market. BigN ML product teams will expand and stay as well paid as they are. Then there will be a huge drop as we move to offshore ML product teams that are viewed as cost centers by the remainder of 99% companies. These will be most of the jobs available.

In such a system, a pure ML scientist (usually a PhD) will only exist at the top companies. So if you are not in the top 1-3% percentile, you will not have a pure ML job. However, there will still be hybrid DS-SDE jobs (ML engineering, ML product maintenance, ML-as-a-service user) or hybrid DS-PM jobs (Analysts, Consultants, data driven business decision makers). So, anyone who is not in that top 1-3% will have to pivot to one of these 3 roles.

______________

I won't call this an AI winter. But, it will definitely become boring majority of those employed in the field..

Re: Machine Learning: The Great Stagnation

#33

Post by alexilliamson »

ipunchghosts wrote: post #22
jarym wrote: post #4

> Academics sacrifice material opportunity costs in exchange for intellectual freedom. Most academics I’ve come across only think they’re doing this. My perception is they are too insecure about their self-worth to pursue material opportunities. I admit, the number of academic types I know is not vast so maybe it’s too small a subset to make any judgments

I went a very uncommon route in my career: I started phd school after 14 years of industry. My motivation was I was tired of creating new and interesting things in industry only to have them be killed by politics. So, my idea never saw the light of day san a few dozen people. My thought was that if i have a phd, I am rewarded in my career for publishing these thoughts. If they are published, maybe some organization s…

How has the experience been getting a PhD after so long out of school? I'm quickly approaching 14 years in industry but still haven't ruled out a PhD.

Re: Machine Learning: The Great Stagnation

#34

Post by Tenoke »

Rochus wrote: post #28
Tenoke wrote: post #21

Earlier quoted context omitted.

Gary Marcus has been proven wrong on claims he's made and is mostly just nay-saying with adhoc reasoning made up to support it. >appears less biased than others who benefit from the current high level of investment in DNN. Yes, instead he blatantly tries to benefit from the counter-investment in AI skepticism.

There somehow seems to be an unwritten law of the Python generation like "Thou shalt not criticise Machine Learning". Or is there a better explanation for the emotions that flare up every time someone dampens the exaggerated expectations and reminds us of earlier research in the field of linguistics or AI?

There's plenty of good criticism of given products, papers, and approaches. There's also bad criticism though and it's important to distinguish the two.

Re: Machine Learning: The Great Stagnation

#35

Post by mettamage »

knuthsat wrote: post #9

There's a nice talk by Yann LeCun where he goes on to explain really well why deep learning has such fast progress. [0] He goes on to explain how theory always comes later. I thought that information theory came before practice but turns out it also came after. (there was a bunch of heuristics for sending messages with teletypes) A nice example is Roman technological advances in architecture and materials that predat…

So in that sense forming a theory about something is integrating all the practical stuff you've seen before. To me that make sense, IMO that "practical stuff" are what experiments are, aren't they? Probably the experiments are far from perfect, if remotely good even, but that's how empiricism should work? I think? (hmm... not entirely sure)

Re: Machine Learning: The Great Stagnation

#36

Post by superbcarrot »

otabdeveloper4 wrote: post #19
superbcarrot wrote: post #12

Earlier quoted context omitted.

That's unlikely. There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. The worst case for the field is that progress slows down, people realise that their expectations were unrealistic and the hype inevitably dies down. Which has to happen eventually. So even if ML isn't the hottest thing or a massively growing field, it will still be us…

> There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. If you could name three of them I'd be really grateful. Serious question; everything surrounding ML seems to be only good for (non-monetizable) art projects. As art it is amazing, not going to lie, but "commercial use" seems like a huge stretch.

I think you are underestimating how many of the services you already use incorporate machine learning somewhere. It's not about GANs or deepfakes (right now those are closer to the art projects you are referring to). Some examples for actual applications or classes of applications are image recognition, speech recognition, recommender systems, virtual assistants, fraud detection, financial forecasting.

Re: Machine Learning: The Great Stagnation

#37

Post by mlthoughts2018 »

Ericson2314 wrote: post #25

I still feel like much of AI is a plot to dumb down the modern economy. We want our business people to be just as effective as our quants; we want nothing to require real intellectual labor. The idea that you traditionally have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and all the more so because you are utterly dependent…

> “ The idea that you have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and this breaks their control over the means of production.”

That is a very bizarre description of ML engineers. In every company I’ve worked at, ML is a team or teams that partners with product managers and other engineering teams to learn about problems they need solved. It’s very systematic, boring, and tied heavily to those other teams as the leaders and decision makers.

First you look for high level value propositions, like automating a decision process, removing a customer friction point, creating key metrics where simple metrics are intractable, or various multi-modal information retrieval goals.

You identify opportunities in these kinds of high level areas in lock-step with product managers and other engineering leaders. Then you move on to identify sources of data that can be leveraged, and eventually (much later) you get to the smaller set of work training a model, validating with acceptance tests and hardening the implementation for safe production deployment.

If people are spouting “mumbo jumbo” and making big ML model decisions without lock-step synchronization with other stakeholders, that sounds like organizational dysfunction, not any type of issue with ML.

I also find it odd that you bring up “cost[ing] a lot of money” .. that’s very out of place among everything else mentioned. That seems more like insecurity or jealousy over the market demand for ML talent, and wanting to cut other people down rather than work with them or acknowledge the level of effort it required to get that level of expertise in ML.

Re: Machine Learning: The Great Stagnation

#38

Post by funcDropShadow »

Ericson2314 wrote: post #25

I still feel like much of AI is a plot to dumb down the modern economy. We want our business people to be just as effective as our quants; we want nothing to require real intellectual labor. The idea that you traditionally have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and all the more so because you are utterly dependent…

> Now, that's not to say I am against labor saving devices.

That's good to know. Because "labor saving devices" is by no means the field of machine learning. A hammer is a "labor saving device" if all you have are rocks. I've got the impression that a lot of people conflate machine learning and robotics with labor saving in general. Of course compared to the state of art machine learning let's us hope to find magical shortcuts to get our work done. But an IDE, a word processor, or a compiler is also a labor saving devices. As well as a piece of paper, it is way faster to doodle on a piece of paper at your desk than finding the next cave to doodle.

Re: Machine Learning: The Great Stagnation

#39

Post by magicsmoke »

I dropped out of a PhD in RFIC/MMIC design. Gradient student descent and lack of first principles reasoning hits too close to home. Maybe it's just the nature of highly empirical engineering disciples to just throw many hours of grad student and simulation time blindly at problems in hopes that one lucky fella finds a new optimization.
Post reply on HN