Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning: The Great Stagnation

marksaroufim.substack.com

Post reply on HN
Print view View original

201–210 of 227 posts

Re: Machine Learning: The Great Stagnation

#201

Post by p1esk »

Componica wrote: post #199
p1esk wrote: post #179

Earlier quoted context omitted.

I don't know where that "general consensus" came from, because in 1998 Lecun clearly showed [1] convnets beating all other models, including SVMs, at image recognition. [1] http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf

It like other neural network research was ignored because neural networks were considered a dead-end at the time. In the early 2000s I recreated his LeNet-5 implementation, and no one was interested despite the great results I was getting in OCR and medical image processing with tumor detection. Younger people don't realize there was strong bias against using neural networks in the late 90s up until Hinton's talk on…

Most research will be ignored if it does not have good results, regardless of whether it's novel, or whether it's DL. In fact, a lot of recently published DL research is ignored for exactly this reason. Top DL conferences are so competitive currently that the quality bar is pretty high.

There are lots of ideas floating around in AI field. Some of them might be good, most are not. If you have an idea and want others to look at it you better demonstrate how it outperforms every other method when applied to some task.

Re: Machine Learning: The Great Stagnation

#202

Post by sgt101 »

joe_the_user wrote: post #185
sgt101 wrote: post #177

Earlier quoted context omitted.

>ing is great, has made important and significant strides, it's not a yet science. It involves essentially a series of sophisticated, mathematically informed recipes for feeding data to giant algorithms and having them create something useful (maybe very useful but still). So... scientists need to do scienceing until it is. This is what happened with Biology over the last 50 years after 2000 years of pinning things o…

My original point was in response to OP's saying that not every paper needs to be brilliant. The thing about "normal science" is that begins with more or less verified theories and extends their theory and practice - until they fail, reach the edge of the theories, and the science community must search for alternative hypotheses. Machine learning is more working with a bunch of practices, rules of thumb and suggestio…

Yeah - SVM were considered to be optimal, then they got smashed by deep networks, now people keep bringing them back and saying "they are just as good" and yet deep networks keep being used to do all the breakthrough work.

Re: Machine Learning: The Great Stagnation

#203

Post by BlueTemplar »

Richard Feynman — 'Philosophy of science is about as useful to scientists as ornithology is to birds.'

Well, this blogpost shows where this analogy breaks down : birds (AFAWK) don't try to discuss ornithology.

"bureaucrats running the asylum" is normal science !

Thomas Kuhn has shown how it works in The Structure of Scientific Revolutions :

https://samzdat.com/2018/05/19/science-under-high-modernism/

The problem is that without Kuhn our expectation are set by pop history of science, which only remembers 'anormal', extraordinary science : the paradigm changes.

(Otherwise, this is a great blogpost.)

Re: Machine Learning: The Great Stagnation

#204

Post by BlueTemplar »

p1esk wrote: post #201
Componica wrote: post #199

Earlier quoted context omitted.

It like other neural network research was ignored because neural networks were considered a dead-end at the time. In the early 2000s I recreated his LeNet-5 implementation, and no one was interested despite the great results I was getting in OCR and medical image processing with tumor detection. Younger people don't realize there was strong bias against using neural networks in the late 90s up until Hinton's talk on…

Most research will be ignored if it does not have good results, regardless of whether it's novel, or whether it's DL. In fact, a lot of recently published DL research is ignored for exactly this reason. Top DL conferences are so competitive currently that the quality bar is pretty high. There are lots of ideas floating around in AI field. Some of them might be good, most are not. If you have an idea and want others t…

OP's point is that research is/was being ignored despite having good results. But this is normal, it just takes time, and a critical mass of good results for most researchers to switch to the new paradigm. (A decade is a very short time in the history of science.)

Re: Machine Learning: The Great Stagnation

#205

Post by BlueTemplar »

joe_the_user wrote: post #185
sgt101 wrote: post #177

Earlier quoted context omitted.

>ing is great, has made important and significant strides, it's not a yet science. It involves essentially a series of sophisticated, mathematically informed recipes for feeding data to giant algorithms and having them create something useful (maybe very useful but still). So... scientists need to do scienceing until it is. This is what happened with Biology over the last 50 years after 2000 years of pinning things o…

My original point was in response to OP's saying that not every paper needs to be brilliant. The thing about "normal science" is that begins with more or less verified theories and extends their theory and practice - until they fail, reach the edge of the theories, and the science community must search for alternative hypotheses. Machine learning is more working with a bunch of practices, rules of thumb and suggestio…

> An analogy from a couple years ago is bridge building before physics. You accumulate rules of thumb, you get a vague understand what works. You get better. But you aren't producing a systematic field.

This would be more like Kuhn's pre-paradigmatic (pre-scientific) activity, than moving from one scientific paradigm to another ?

https://samzdat.com/2018/05/19/science-under-high-modernism/

(I don't know which one is more appropriate here.)

Re: Machine Learning: The Great Stagnation

#206

Post by BlueTemplar »

tracyhenry wrote: post #107
b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

> It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). Is the deep learning really the result of incremental research? The SOTA chasing frenzy comes after the discovery of deep learning. The motivation of incremental research can hardly be justified as to discover the next deep learning, although they might do.

What does SOTA means ? "State of the Art" ?

Re: Machine Learning: The Great Stagnation

#207

Post by BlueTemplar »

spullara wrote: post #186
b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

I'd argue that deep learning isn't even new. I was doing multi-layer neural networks back in 1992. The compute power though was woefully underpowered to actually do what they are using it for today.

Yeah, "multilayer perceptrons" date back to the 80's...

Re: Machine Learning: The Great Stagnation

#208

Post by BlueTemplar »

SilkStocking wrote: post #132
JPKab wrote: post #124

Earlier quoted context omitted.

Just to be clear, there's a lot of researchers making money that aren't producing squat. There are many AI ethicists I admire, and there are many that can't code, can't produce, and just spend their time getting into arguments with people like Yann LeCunn, who actually have produced groundbreaking research.

LeCon's work wasn't groundbreaking; he was saved from oblivion only by Moore's Law. LeCun stood still doing the same thing for decades. But when wave after wave of faster hardware arrived he was finally declared successful. So if you want to credit success to dumb luck, or to stubbornness, or insanity (as Einstein said: "doing the same thing over and over again and expecting different results." then LeCon is your man…

> "Well, I knew the hardware would speed up and then my neural net software would work exactly as I designed it to and manifest AI. So I waited twenty years."

You're talking like we only discovered Moore's law now, instead of it being a good bet for the last half-century...

Re: Machine Learning: The Great Stagnation

#209

Post by zwaps »

deeeeplearning wrote: post #162
SoSoRoCoCo wrote: post #115

This article is dead-on, but I think it is missing a fairly large segment of where ML is actually working well: anomaly detection and industrial defect detection. While I agree that everyone was shocked, myself included, when we saw how well SSD and YOLO worked, the last mile problem is stagnating. What I mean is: 7 years ago I wrote an image pipeline for a company using traditional AI methods. It was extremely chall…

>The industry is stagnated for exactly the reasons brought up: we don't know how to squeeze out the last mile problem because NNs are EFFING HARD and research is very math heavy: e.g., it cannot be hacked by a Zuck-type into a half-assed product overnight, it needs to be carefully researched for years. This makes programmers sad, because by nature we love to brute force trial-and error our code, and homey don't play…

And yet, it might fail in actual application.

If every DGP could be captured by fast-tuning a sophisticated enough model, science probably would be solved even before DL.

Re: Machine Learning: The Great Stagnation

#210

Post by andrewon »

joe_the_user wrote: post #168
b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). In no way is machine learning unique in this. Except ... while machine learning is great, has made important and significant strides, it's not a yet science. It involves essentially a series of sophisticated, mathematic…

Currently ML is building model to fit data and trying to avoid over-fitting. I think "engineering field" is a more suitable term.
Post reply on HN