Machine Learning: The Great Stagnation • Hacker News Board

Re: Machine Learning: The Great Stagnation

#211

Post by b3kart » Fri, Jan 15, 2021, 2:29 PM UTC

BlueTemplar wrote: post #206

tracyhenry wrote: post #107

Earlier quoted context omitted.

> It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). Is the deep learning really the result of incremental research? The SOTA chasing frenzy comes after the discovery of deep learning. The motivation of incremental research can hardly be justified as to discover the next deep learning, although they might do.

What does SOTA means ? "State of the Art" ?

Yes. Alternatively, “look, ma, my number is the biggest number”. Apologise for the sass, just not very happy that SOTA chasing is often discouraging original contributions that would actually move the field forward.

Re: Machine Learning: The Great Stagnation

#212

Post by b3kart » Fri, Jan 15, 2021, 2:33 PM UTC

joe_the_user wrote: post #185

sgt101 wrote: post #177

Earlier quoted context omitted.

>ing is great, has made important and significant strides, it's not a yet science. It involves essentially a series of sophisticated, mathematically informed recipes for feeding data to giant algorithms and having them create something useful (maybe very useful but still). So... scientists need to do scienceing until it is. This is what happened with Biology over the last 50 years after 2000 years of pinning things o…

My original point was in response to OP's saying that not every paper needs to be brilliant. The thing about "normal science" is that begins with more or less verified theories and extends their theory and practice - until they fail, reach the edge of the theories, and the science community must search for alternative hypotheses. Machine learning is more working with a bunch of practices, rules of thumb and suggestio…

Thanks for your comment. I should go and read someone who’s actually studied the phenomenon, instead of making conjectures.

Re: Machine Learning: The Great Stagnation

#213

Post by deeeeplearning » Fri, Jan 15, 2021, 4:09 PM UTC

zwaps wrote: post #209

deeeeplearning wrote: post #162

Earlier quoted context omitted.

>The industry is stagnated for exactly the reasons brought up: we don't know how to squeeze out the last mile problem because NNs are EFFING HARD and research is very math heavy: e.g., it cannot be hacked by a Zuck-type into a half-assed product overnight, it needs to be carefully researched for years. This makes programmers sad, because by nature we love to brute force trial-and error our code, and homey don't play…

And yet, it might fail in actual application. If every DGP could be captured by fast-tuning a sophisticated enough model, science probably would be solved even before DL.

We're talking about building useable products not solving science...

Re: Machine Learning: The Great Stagnation

#214

Post by deeeeplearning » Fri, Jan 15, 2021, 4:10 PM UTC

causalmodels wrote: post #194

deeeeplearning wrote: post #162

Earlier quoted context omitted.

>The industry is stagnated for exactly the reasons brought up: we don't know how to squeeze out the last mile problem because NNs are EFFING HARD and research is very math heavy: e.g., it cannot be hacked by a Zuck-type into a half-assed product overnight, it needs to be carefully researched for years. This makes programmers sad, because by nature we love to brute force trial-and error our code, and homey don't play…

Fast.ai works pretty well when you're working on standard tasks but starts to fall apart when you want to do something more exotic.

I doubt there's much in CV for instance that couldn't be achieved easily with fast.ai. You don't need to be doing exotic things to build a product.

Re: Machine Learning: The Great Stagnation

#215

Post by zwaps » Fri, Jan 15, 2021, 8:21 PM UTC

deeeeplearning wrote: post #213

zwaps wrote: post #209

Earlier quoted context omitted.

And yet, it might fail in actual application. If every DGP could be captured by fast-tuning a sophisticated enough model, science probably would be solved even before DL.

We're talking about building useable products not solving science...

My point being that the reason many products end up not usable, ref. accounts in this thread, is the same reason why science isn’t solved and doing ML correctly isn’t easy.

Re: Machine Learning: The Great Stagnation

#216

Post by causalmodels » Fri, Jan 15, 2021, 8:59 PM UTC

deeeeplearning wrote: post #214

causalmodels wrote: post #194

Earlier quoted context omitted.

Fast.ai works pretty well when you're working on standard tasks but starts to fall apart when you want to do something more exotic.

I doubt there's much in CV for instance that couldn't be achieved easily with fast.ai. You don't need to be doing exotic things to build a product.

Never said you need an exotic model to build a product nor that you couldn't do exotic things in fast.ai. Fast.ai is just a leaky abstraction.

Re: Machine Learning: The Great Stagnation

#217

Post by gbolcer » Fri, Jan 15, 2021, 10:08 PM UTC

gbolcer wrote: post #196

xpe wrote: post #143

Earlier quoted context omitted.

> I have this saying: all machine learning is just clever tricks ... It appears to me that you are missing the mark here, unless this is largely a definitional issue. Do you consider the foundations of ML to be a clever trick? Do you think human brains primarily learn by clever tricks? When a metaphor or saying falls apart with one more level of questioning, I would suggest it may be time to find a better metaphor.

To your first question, yes it's just a style of solving a problem that can be implemented in software for 90%+ of the time. For your second, I think that's part of the problem. Most people confuse how a human brain learns with what is actually running in a machine learning program. It's similar at some level, but not really doing the same thing at another.

BTW, Google Brain posted this on the subject of "brute force" solutions yesterday. I thought it would be of interest here.

The big idea here is that enough brute force will lead to better compute-use techniques which will in turn make it possible to do more with less compute. But the current reality is that these systems don’t tend to justify their existence when compared to greener, more useful technologies. It’s hard to pitch an AI system that can only be operated by trillion-dollar tech companies willing to ignore the massive carbon footprint a system this big creates.

Re: Machine Learning: The Great Stagnation

#218

Post by jabl » Sat, Jan 16, 2021, 1:37 PM UTC

counters wrote: post #98

AndrewKemendo wrote: post #80

Earlier quoted context omitted.

>We've been in an exciting deep learning craze for a while, but it's silly to expect it to last forever. Back to the grind now. This was effectively my response to hardmaru when this topic came up on reddit [1] Basically 2010-2018 was an open field for ML/DL research with old(ish) methods being rapidly applied to low hanging fruit and large datasets with newly cheap compute. Deepmind and others are actually making ne…

> Basically 2010-2018 was an open field for ML/DL research with old(ish) methods being rapidly applied to low hanging fruit and large datasets with newly cheap compute. Absolutely! And what many folks need to continue to remember is that many scientific disciplines and domains are really just starting to wrestle with the utility and implications of this first generation of deep learning tools and applications. I grad…

This seems to be the trend in every computational science field. Take something compute intensive, apply DL magic and get results in a fraction of the CPU time as the old approach.

Similarly on the commercial side, plenty of opportunity to solve existing problems with DL.

So while it may be true that DL research progress has slowed, still plenty to do in applying existing DL.

Re: Machine Learning: The Great Stagnation

#219

Post by jabl » Sat, Jan 16, 2021, 1:39 PM UTC

BlueTemplar wrote: post #204

p1esk wrote: post #201

Earlier quoted context omitted.

Most research will be ignored if it does not have good results, regardless of whether it's novel, or whether it's DL. In fact, a lot of recently published DL research is ignored for exactly this reason. Top DL conferences are so competitive currently that the quality bar is pretty high. There are lots of ideas floating around in AI field. Some of them might be good, most are not. If you have an idea and want others t…

OP's point is that research is/was being ignored despite having good results. But this is normal, it just takes time, and a critical mass of good results for most researchers to switch to the new paradigm. (A decade is a very short time in the history of science.)

Like they say, science progresses one funeral at a time.

Re: Machine Learning: The Great Stagnation

#220

Post by xpe » Sat, Jan 16, 2021, 6:36 PM UTC

gbolcer wrote: post #196

xpe wrote: post #143

Earlier quoted context omitted.

> I have this saying: all machine learning is just clever tricks ... It appears to me that you are missing the mark here, unless this is largely a definitional issue. Do you consider the foundations of ML to be a clever trick? Do you think human brains primarily learn by clever tricks? When a metaphor or saying falls apart with one more level of questioning, I would suggest it may be time to find a better metaphor.

To your first question, yes it's just a style of solving a problem that can be implemented in software for 90%+ of the time. For your second, I think that's part of the problem. Most people confuse how a human brain learns with what is actually running in a machine learning program. It's similar at some level, but not really doing the same thing at another.

> To your first question, yes it's just a style of solving a problem that can be implemented in software for 90%+ of the time.

What insight is gained by this statement? What does it explain? What does it downplay?

I'm not currently seeing much value in it. I'll explain why. Saying 'just a method of solving a problem' comes across as reductive without being useful.

Imagine if someone said 'flying is just a means of movement' in the context of studying a hummingbird's agility. It says more about the speaker than the subject. It suggests the person is uninterested or focused on other things.

So with regards to your statement, it suggests you don't care and/or don't appreciate what makes learning difficult.

I'm talking about learning theory. About generalization given data. This is certainly not easy. Yes, it can be encoded in an algorithm, but that does not make it less interesting.