Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning: The Great Stagnation

marksaroufim.substack.com

Post reply on HN
Print view View original

171–180 of 227 posts

Re: Machine Learning: The Great Stagnation

#171

Post by refactor_master »

joe_the_user wrote: post #168
b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). In no way is machine learning unique in this. Except ... while machine learning is great, has made important and significant strides, it's not a yet science. It involves essentially a series of sophisticated, mathematic…

> An analogy from a couple years ago is bridge building before physics. You accumulate rules of thumb, you get a vague understand what works. You get better. But you aren't producing a systematic field.

Sounds like science to me; systematic recording of (perceived) cause and effect.

Arguably, deep learning and cell biology both appear like equal parts pure wizardry and flailing in the dark, but maybe that’s just because we haven’t gathered enough pieces yet, and not necessarily because people are doing the wrong things, thus failing to advance?

Re: Machine Learning: The Great Stagnation

#172

Post by tmotwu »

SoSoRoCoCo wrote: post #115

This article is dead-on, but I think it is missing a fairly large segment of where ML is actually working well: anomaly detection and industrial defect detection. While I agree that everyone was shocked, myself included, when we saw how well SSD and YOLO worked, the last mile problem is stagnating. What I mean is: 7 years ago I wrote an image pipeline for a company using traditional AI methods. It was extremely chall…

I worked and built out a proof of concept industrial defect detection system recently, with a large focus on modern DNN architectures. We worked with a plant to curate a 30000+ multi-class defect dataset, many with varying lighting and environment conditions. As you said, modifying and parameter tuning NN is not always a hopeful endeavor.

However, you can make significant gains to your models by going back to traditional image filtering/augmentation. Sticking with well researched object detectors/segmentation algorithms and putting our effort on improving the algorithms that cleans up the data takes you far. It's impossible to avoid because images will always be full of reflections, artifacts, strange coloration unless you have the perfect lighting tunnel setup; doable nonetheless.

Re: Machine Learning: The Great Stagnation

#173

Post by dhairya »

alexilliamson wrote: post #33
ipunchghosts wrote: post #22

Earlier quoted context omitted.

I went a very uncommon route in my career: I started phd school after 14 years of industry. My motivation was I was tired of creating new and interesting things in industry only to have them be killed by politics. So, my idea never saw the light of day san a few dozen people. My thought was that if i have a phd, I am rewarded in my career for publishing these thoughts. If they are published, maybe some organization s…

How has the experience been getting a PhD after so long out of school? I'm quickly approaching 14 years in industry but still haven't ruled out a PhD.

I just started my PhD in AI (specifically NLP). Also took a non-traditional route (creative writing major -> 10 years in industry -> phd).

It's interesting so far. Research feels very open ended compared to industry. While I was in industry (AI fintech startup), even though goals were rapidly changing, I had a good idea of what problems to work on and how to gauge progress.

In contrast, research is far more undefined. There are days I feel lost and other where I'm chasing rabbits down deep holes. It's been hard for me to figure out if the problems I'm trying to solve are worth exploring (are they good research question? and more importantly are they publishable).

But that being said, it's only about 6 months in and I feel like I'm still learning what it means to do research. I've definitely enjoyed having the space to explore problems at my own pace and think deeply about them.

Re: Machine Learning: The Great Stagnation

#174

Post by YeGoblynQueenne »

Rochus wrote: post #28
Tenoke wrote: post #21

Earlier quoted context omitted.

Gary Marcus has been proven wrong on claims he's made and is mostly just nay-saying with adhoc reasoning made up to support it. >appears less biased than others who benefit from the current high level of investment in DNN. Yes, instead he blatantly tries to benefit from the counter-investment in AI skepticism.

There somehow seems to be an unwritten law of the Python generation like "Thou shalt not criticise Machine Learning". Or is there a better explanation for the emotions that flare up every time someone dampens the exaggerated expectations and reminds us of earlier research in the field of linguistics or AI?

Let's not make this a generational divide thing, shall we? There's people of all ages who seem to think AI research is like a silly ball game, where one cheers their home team's plays and boos their er, whatsitcalled, the other team's plays, no matter the plays.

And who can blame them? AI (read: machine learning) (read: deep learning) research has turned into a huge feeding frenzy. People see all the billions thrown about by Google, Facebook et al. and they go crazy. Maybe they think that if they cheer hard enough and boo hard enough they'll look knowledgeable and "passionate about machine learning" and maybe someone will hire them. Maybe they just want to be on the right side of history, with the winners, not the losers. And when there's so much money to win, there sure are plenty of losers!

A while ago someone posted here an article that advised that to become expert in machine learning one should (among other things) "flashcard X papers in major sub-fields" or something along those lines. Pretty revealing of what people are thinking of: Google is hiring machine learning specialists. Shut up Garry Marcus, you'll scare the fish off.

Re: Machine Learning: The Great Stagnation

#175

Post by b3kart »

joe_the_user wrote: post #168
b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). In no way is machine learning unique in this. Except ... while machine learning is great, has made important and significant strides, it's not a yet science. It involves essentially a series of sophisticated, mathematic…

I would agree that as a science machine learning is in its infancy. There's statistical learning theory, but arguably that's a fairly small subfield, if not a separate field completely. ML has largely been an engineering enterprise so far, but in my experience the interest in developing underlying theory has ramped up in recent years.

What we should be careful about is to not be too strict when defining "science". In my view the goal of any scientific field is to build understanding that is useful for predicting outcomes of experiments. Now, this understanding could be defined mathematically, but it doesn't have to be. Don't see why building heuristics can't be a part of this, assuming such heuristics reliably predict outcomes of experiments.

Re: Machine Learning: The Great Stagnation

#176

Post by lacker »

p1esk wrote: post #145
b3kart wrote: post #141

Earlier quoted context omitted.

>they don't work. In the same way that ANNs didn't work for quite some time, until we've had the compute and the data to train them successfully? I get that it's important to prove that an idea is worthwhile, and the easiest way to do that is to use it to solve a practical problem. At the same time, I am conscious that we shouldn't put all our eggs in the deep learning basket: who knows where the ceiling is going to…

No, not in the same way. Convnets and lstms worked great when they were invented in the 90s. Yes, they were limited by the available compute power but they did work well right away.

When I was in grad school in 1999 studying AI the general consensus was that neural networks didn't work very well, and that you were better off with the more mathematically grounded methods like support vector machines. TD-Gammon was just about the only success story for neural networks, there hadn't been much else since 1992.

Re: Machine Learning: The Great Stagnation

#177

Post by sgt101 »

joe_the_user wrote: post #168
b3kart wrote: post #43

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning).…

There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). In no way is machine learning unique in this. Except ... while machine learning is great, has made important and significant strides, it's not a yet science. It involves essentially a series of sophisticated, mathematic…

>ing is great, has made important and significant strides, it's not a yet science. It involves essentially a series of sophisticated, mathematically informed recipes for feeding data to giant algorithms and having them create something useful (maybe very useful but still).

So... scientists need to do scienceing until it is. This is what happened with Biology over the last 50 years after 2000 years of pinning things on cards and putting them in draws.

Re: Machine Learning: The Great Stagnation

#178

Post by p1esk »

dougabug wrote: post #160
p1esk wrote: post #145

Earlier quoted context omitted.

No, not in the same way. Convnets and lstms worked great when they were invented in the 90s. Yes, they were limited by the available compute power but they did work well right away.

ConvNets were invented in the 80s.

The first convnet that is similar to what we use today was described in 1989 [1], and immediately became the best method to do a handwritten digit recognition.

[1] http://yann.lecun.com/exdb/publis/pdf/lecun-89e.pdf

Re: Machine Learning: The Great Stagnation

#179

Post by p1esk »

lacker wrote: post #176
p1esk wrote: post #145

Earlier quoted context omitted.

No, not in the same way. Convnets and lstms worked great when they were invented in the 90s. Yes, they were limited by the available compute power but they did work well right away.

When I was in grad school in 1999 studying AI the general consensus was that neural networks didn't work very well, and that you were better off with the more mathematically grounded methods like support vector machines. TD-Gammon was just about the only success story for neural networks, there hadn't been much else since 1992.

I don't know where that "general consensus" came from, because in 1998 Lecun clearly showed [1] convnets beating all other models, including SVMs, at image recognition.

[1] http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf

Re: Machine Learning: The Great Stagnation

#180

Post by njanirudh »

SilkStocking wrote: post #132
JPKab wrote: post #124

Earlier quoted context omitted.

Just to be clear, there's a lot of researchers making money that aren't producing squat. There are many AI ethicists I admire, and there are many that can't code, can't produce, and just spend their time getting into arguments with people like Yann LeCunn, who actually have produced groundbreaking research.

LeCon's work wasn't groundbreaking; he was saved from oblivion only by Moore's Law. LeCun stood still doing the same thing for decades. But when wave after wave of faster hardware arrived he was finally declared successful. So if you want to credit success to dumb luck, or to stubbornness, or insanity (as Einstein said: "doing the same thing over and over again and expecting different results." then LeCon is your man…

Are you telling me CNN are not groundbreaking?
Post reply on HN