Why is machine learning 'hard'? (2016)
ai.stanford.edu
Why is machine learning 'hard'? (2016)
1–10 of 144 posts
Re: Why is machine learning 'hard'? (2016)
#2"An aspect of this difficulty involves building an intuition for what tool should be leveraged to solve a problem."
While I agree with the good point about debugging, like many others, I am rather worried that we're increasingly deploying AI/ML where we shouldn't be deploying it. Hence, the above quote.
Re: Why is machine learning 'hard'? (2016)
#3Discussed at the time:
Why is machine learning ‘hard’? - https://news.ycombinator.com/item?id=12936891 - Nov 2016 (88 comments)
Re: Why is machine learning 'hard'? (2016)
#4Discussed at the time: Why is machine learning ‘hard’? - https://news.ycombinator.com/item?id=12936891 - Nov 2016 (88 comments)
Love that thread. The top comment is excellent:
> Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best".
> To me, that feels weird and worrying. It's like we don't yet understand ML properly yet to definitively say, for a given data set, what sort of model we'll need.
This embodies the very fundamental difference between science and engineering. With science, you make a discovery, but rarely do we ask "what was the magical combination that let me find the needle in the haystack today?" We instead just pass on the needle and show everyone we found it.
Should we work on finding out the magic behind hyperparameters? In bioinformatics, the brilliant mathematician Lior Pachter once attacked the problem of sequence alignment using the tools of tropical algebra: what parameters to the alignment algorithms resulted in which regimes of solutions? It was beautiful. It was great to understand. But I'm not sure if it even ever got published (though it likely did). Having reasonable parameters is more important than understanding how to pick them from first principles, because even if you know all the possible output regimes for different segments of the hyper parameter space, really the only thing we care about is getting a functionally trained model at the end.
Sometimes deeper understanding provides deeper insights to the problems at hand. But often, they don't, even when the deeper understanding is beautiful. If the hammer works when you hold it a certain way, that's great, but understanding all possible ways to hold a hammer doesn't always help get the nail in better.
Re: Why is machine learning 'hard'? (2016)
#5I'm a bit sceptical of the exponentially harder debugging claim.
First it looks polynomially harder for the given example :p.
Second other engineering domains arguably have additional dimensions which correspond to the machine learning ones mentioned in the article. The choice of which high level algorithm to implement is another dimension to traditional software engineering that seemingly exists and corresponds to the model dimension. This is often codified as 'design'.
The data dimension often exists as well in standard learning software engineering. [Think of a system that is 'downstream' of other].
It's probably a lot simpler to deal with these dimensions in standard software engineering - but then this is what makes machine learning harder, not that there are simply 'more dimensions'.
The delayed debugging cycles point seems a lot more valid.
Re: Why is machine learning 'hard'? (2016)
#6Discussed at the time: Why is machine learning ‘hard’? - https://news.ycombinator.com/item?id=12936891 - Nov 2016 (88 comments)
Love that thread. The top comment is excellent: > Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". > To me, that feels weird and worrying. It's like we don't yet un…
The hammer analogy doesn't make much sense because for a hammer we can actually use our scientific knowledge to compute the best possible way to hold the tool, and we can make instruments that are better than hammers, like pneumatic hammers, pile drivers, etc.
With your argument, we would be stuck with the good old, but basic hammer for the rest of time.
Re: Why is machine learning 'hard'? (2016)
#7The #1 thing that makes it ‘hard’ in real life is that nobody wants to make training and test sets. So we have 50,000 papers on the NIST digits but no insight into ‘would this work for a different problem?’ (Ironically the latter might have been exactly what academics would have needed to understand why these algorithms work!)
Re: Why is machine learning 'hard'? (2016)
#8there are uncountable sets all over the place, and in practical terms, the repl loop may have a week long training lag after you hit enter.
also, the data is almost always complete shit.
lol. there’s no mystery why it’s hard.
Re: Why is machine learning 'hard'? (2016)
#9>> It becomes essential to build an intuition for where something went wrong based on the signals available.
This has always been my approach. I learned programming way before I had access to debuggers and other methods to dig in, set breakpoints and step through code to see where it was going wrong. As a result, when I got in the real world I kind of looked down on people using those tools (mostly because I hate tools actually). But then I saw people get to the root of problems that I don't think I ever could have solved, and I started to appreciate those tools and the detail you could get to. My preference is still to have a great understanding of how algorithms work, how the code is written, and what the problem is, and noodle out what and where things may be going wrong. I only switch to detailed monitoring of the insides when "thinking about it" fails. Maybe I should have gone into this ML stuff ;-)
Re: Why is machine learning 'hard'? (2016)
#10"An aspect of this difficulty involves building an intuition for what tool should be leveraged to solve a problem." While I agree with the good point about debugging, like many others, I am rather worried that we're increasingly deploying AI/ML where we shouldn't be deploying it. Hence, the above quote.
I’m old enough to have learned that the secret to success is much less knowing the tool of the moment than picking the right tool for a job.
The right tool may in fact be the new one, and LLM do open a lot of doors with zero shot capabilities, but oftentimes they can underperform a well tuned heuristic. It’s the ability to pick the right tool that is key.