Machine Learning: The Great Stagnation • Hacker News Board

Re: Machine Learning: The Great Stagnation

#151

Post by oxinabox » Thu, Jan 14, 2021, 7:12 PM UTC

JHonaker wrote: post #110

I kind of agree with the author's major sentiment: that ML research is stuck in a rut with incremental improvement. However, the longer the article goes on, the less and less I agree with any of their statements. They start of criticizing the incremental improvers. They advocate later that if "stack more layers" beats a method, the method isn't good while completely ignoring anything other than the standard SOTA metr…

> and there are some great AD packages, but in no way is everything automatically differentiable (even with the nice packages), nor is that a design goal.

I work on the AD infastructure for Julia. That absolutely is a design goal. Certainly we are not there yet; we still have a long way to go. But that is where we want to go to.

With the cavet that thigns that are not mathematically defined to have derivatives (e.g. the derivative of `xs[i]` with respect to `i`) we won't differentiate those.

But for stuff like mutation (the big thing Zygote doesn't support (though some of our other ADs do)), we sure do want it to.

Re: Machine Learning: The Great Stagnation

#152

Post by karmasimida » Thu, Jan 14, 2021, 7:23 PM UTC

Matrix multiplication is nothing to be laughed at. It is how anyone performs linear projection anyway.

Fair enough to say, currently ML research is fixated on exhausting combinations of blocks to squeeze a marginal improve on a few public benchmarks.

But it is like saying software has been stagnated since we are not getting new ISA created. Building ML application remains as challenging as ever, it is just modeling itself has become streamlined before everything else.

Re: Machine Learning: The Great Stagnation

#153

Post by kkylin » Thu, Jan 14, 2021, 7:35 PM UTC

dcow wrote: post #142

kkylin wrote: post #130

Earlier quoted context omitted.

I think you are exactly right, and want to add a couple things ( ): - It's easy to recognize scientific / technological revolutions in hindsight, but at the time they're anything but. I think most recognize the importance of persistence in the progenitor of an idea. What's often missed in these discussions is how important the subsequent incremental progress is: working out consequences of a theoretical insight, figu…

Where in the world did you come across the ( ) ... ( ) style? I like it.

Thanks. I actually type "(asterisk)", forgetting every single time that HN uses asterisks for formatting, and too lazy to go back and fix it...

Re: Machine Learning: The Great Stagnation

#154

Post by SoSoRoCoCo » Thu, Jan 14, 2021, 8:05 PM UTC

gok wrote: post #144

SoSoRoCoCo wrote: post #115

This article is dead-on, but I think it is missing a fairly large segment of where ML is actually working well: anomaly detection and industrial defect detection. While I agree that everyone was shocked, myself included, when we saw how well SSD and YOLO worked, the last mile problem is stagnating. What I mean is: 7 years ago I wrote an image pipeline for a company using traditional AI methods. It was extremely chall…

> This makes programmers sad, because by nature we love to brute force trial-and error our code, and homey don't play that game with machine learning. Huh? If anything I would say ML is way more trial-and-error focused than imperative programming.

Well, if your only experience is reading python opencv stack overflow posts, then of course...

Re: Machine Learning: The Great Stagnation

#155

Post by wpietri » Thu, Jan 14, 2021, 8:11 PM UTC

Ericson2314 wrote: post #79

wpietri wrote: post #70

Earlier quoted context omitted.

This seems fair to me. The executive view of ML is "can you do me a magic?" And as this article's "Graduate Student Descent" bit makes clear, the worker response is often to semi-randomly perturb code, show some graphs, and say, "Is this a magic?" For me, most software development is about finding something boring and laborious. We get a computer to do the work so humans can level up and work on something requiring a…

> Especially people in the managerial caste, as the reigning dogma there is that management is a universal skill. Details are for the little people. Exactly. Machine learning is the perfect ideological duel. It's "universal labor" for "universal management", and both sides are equally illiterate in the ways of the world.

Ooh, that's very well put.

Re: Machine Learning: The Great Stagnation

#156

Post by klik99 » Thu, Jan 14, 2021, 8:19 PM UTC

It's about time someone formalized Graduate Student Descent, please someone write a paper diving into all it's properties!

"Graduate Student Descent is one of the most reliable ways of getting state of the art performance in Machine Learning today and it’s also ... fully parallelizable"

Re: Machine Learning: The Great Stagnation

#157

Post by gok » Thu, Jan 14, 2021, 8:28 PM UTC

SoSoRoCoCo wrote: post #154

gok wrote: post #144

Earlier quoted context omitted.

> This makes programmers sad, because by nature we love to brute force trial-and error our code, and homey don't play that game with machine learning. Huh? If anything I would say ML is way more trial-and-error focused than imperative programming.

Well, if your only experience is reading python opencv stack overflow posts, then of course...

I mean in the sense that using ML for a problem often requires just trying a dozen different modeling techniques, then a bunch of a hyper-parameter searching, then a bunch of stochastic tuning…

Re: Machine Learning: The Great Stagnation

#158

Post by SoSoRoCoCo » Thu, Jan 14, 2021, 8:41 PM UTC

gok wrote: post #157

SoSoRoCoCo wrote: post #154

Earlier quoted context omitted.

Well, if your only experience is reading python opencv stack overflow posts, then of course...

I mean in the sense that using ML for a problem often requires just trying a dozen different modeling techniques, then a bunch of a hyper-parameter searching, then a bunch of stochastic tuning…

Oh. I see what you mean. Yeah, I guess by definition backwards propagation is trial-end-error. Huh, I never thought of it that way. Thanks for clarifying, I thought you were being saucy: my apologies for being snarky.

Re: Machine Learning: The Great Stagnation

#159

Post by tartaglia » Thu, Jan 14, 2021, 8:51 PM UTC

z77dj3kl wrote: post #97

CyberRabbi wrote: post #93

Interesting article but the part where he says “all” you need to learn are matrix multiplications seems vacuously true to the point of meaninglessness. All numerical methods across every field essentially boil down to matrix multiplications. Your math ability needs to be such that you can transform a problem into a form that can be computed numerically using matrix multiplications, which requires more skill (sometime…

All you need is ZFC...

All you need is ZF...

Re: Machine Learning: The Great Stagnation

#160

Post by dougabug » Thu, Jan 14, 2021, 8:53 PM UTC

p1esk wrote: post #145

b3kart wrote: post #141

Earlier quoted context omitted.

>they don't work. In the same way that ANNs didn't work for quite some time, until we've had the compute and the data to train them successfully? I get that it's important to prove that an idea is worthwhile, and the easiest way to do that is to use it to solve a practical problem. At the same time, I am conscious that we shouldn't put all our eggs in the deep learning basket: who knows where the ceiling is going to…

No, not in the same way. Convnets and lstms worked great when they were invented in the 90s. Yes, they were limited by the available compute power but they did work well right away.

ConvNets were invented in the 80s.