To be fair, chip design has both of these problems to a much greater degree than machine learning:
- Exponentially Difficult Debugging
- Delayed Debugging Cycles
Why is machine learning 'hard'? (2016)
91–100 of 144 posts
Re: Why is machine learning 'hard'? (2016)
#92I used to work on an ML research team. In addition to what the author mentions, there is an entirely separate issue: whether or not what you're attempting to do is possible with the approach you've chosen. Consider making an iOS app. For the most part, an experienced software engineer can tell you if making a given app is possible, and they'll have a relatively clear idea about the steps required to realize the idea.…
I have always thought of ML (not DL) as phenomena that can be modelled mathematically. It turns out that not all problems have a great mathematical model like self driving cars for instance and so the search continues...
All of ML, including DL, are literally implemented using mathematical models. Alas, a model is just a model and doesn’t imply it works well or imply that it’s simple or easily discoverable.
Re: Why is machine learning 'hard'? (2016)
#93I found the explanation quite convoluted (pun intended). Any system with billions of parameters is bound to be difficult to reason about.
Re: Why is machine learning 'hard'? (2016)
#94For me it’s 2 things: math is already hard even with a teacher and after you found a job it’s just you and your screen. The other part is research moves at a crazy speed it’s very hard to keep up I think while having a job.
Re: Why is machine learning 'hard'? (2016)
#95Earlier quoted context omitted.
Here’s an example of something similar. Say you have a baseline model with an AUC of 0.8. There’s a cool feature you’d like to add. After a week or two of software engineering to add it, you get it into your pipeline. AUC doesn’t budge. Is it because you added it in the wrong place? Is the feature too noisy? Is it because the feature is just a function of your existing features? Is it because your model isn’t big eno…
Why do you spend weeks adding something instead of directly testing all the later hypotheses?
In some cases you can directly test hypotheses like that, but more often than not, there isn’t a way to test without just trying.
Re: Why is machine learning 'hard'? (2016)
#96I used to work on an ML research team. In addition to what the author mentions, there is an entirely separate issue: whether or not what you're attempting to do is possible with the approach you've chosen. Consider making an iOS app. For the most part, an experienced software engineer can tell you if making a given app is possible, and they'll have a relatively clear idea about the steps required to realize the idea.…
I think one issue also, is that ML is so large as a field, it ecompasses huge subfields, or related fields (statistics, optimizatiob, etc...)
Re: Why is machine learning 'hard'? (2016)
#97Earlier quoted context omitted.
All of these hypotheses will take on the order of days to check. OK, but you can check them, right? How is that different from a regular software bug?
In software engineering you can test things in something on the order of seconds to minutes. Functions have fixed contracts which can be unit tested. In ML your turnaround time is days. That alone makes things harder. Further, some of the problems I listed are open-ended which makes it very difficult to debug them.
> In software engineering you can test things in something on the order of seconds to minutes. Functions have fixed contracts which can be unit tested.
I think this only applies to a certain subset of software engineering, the one that rhymes with "tine of christmas".
Implementing bitstream formats is an area I'm very familiar with, and I dance when an issue takes seconds to resolve. Sometimes you need to physically haul a vendor's equipment to the lab. In broadcast we have this thing called "Interops" where tons of software and hardware vendors do just this, but in a more convention-esque style (actually is often done at actual conventions).
Re: Why is machine learning 'hard'? (2016)
#98Machine learning isn't comparable to software development. It is a statistical modelling exercise. This is like asking why advertising is hard - if a non-expert wades in to a different domain then they may find it has different challenges than what they are used to! This is just a specific case of the normal things that analysts routinely deal with. The major challenges in this youthful field of machine learning are…
I dont feel like im doing statistical modeling when i do ml. Usually feels more like pipe alignment, followed by tremendous amounts of debugging.
The heuristic I use for distinguishing between statistical modelling, machine learning and AI is is through feature engineering and model specification:
- Statistical modelling: Manual feature engineering, manual model specification (y = ax + b)
- Machine learning: Manual feature engineering, automated model specification (y = ax + b or y = ax^2 + b, I don't care, the algorithm should figure it out).
- AI: Automated feature engineering (e.g. CNN), automated model specification
Re: Why is machine learning 'hard'? (2016)
#99Earlier quoted context omitted.
But aren't all science basically like this? If you know your hypothsis works before you do the experiments, it's not science anymore.
Not really. It’s easy to tell relative to existing methods whether the size of data will solve the problem. For example, if you’re trying to solve a classification problem with a large number of labels, but only have a small amount of training data for some (or all) of them, it will probably never work.
> classification problem with a large number of labels, but only have a small amount of training data for some (or all) of them, it will probably never work
That is true. On the other hand i have seen someone once perform a trick which looked miraculous to me.
We had a classification problem with a small number of labels (~3). And one of the labels had unfortunately way less samples in our training set. Then someone trained a GAN to turn the images of the abundant labels into images of the rare labels. We added those syntetically generated images to the training set and it improved our classification performance as best as we could tell.
That one still feels a bit like black magic to me to be honest. Almost as if we got more out of less with a trick.
Re: Why is machine learning 'hard'? (2016)
#100What do we even mean by "Machine Learning"?
Understanding ML algorithms and the math behind, being able to change the algorithms and devise new ones?
Using some ML libraries and frameworks?
Taking an implemented solution and training it and fine tuning parameters?