Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is machine learning 'hard'? (2016)

ai.stanford.edu

Post reply on HN
Print view View original

21–30 of 144 posts

Re: Why is machine learning 'hard'? (2016)

#21

Post by epistasis »

ummonk wrote: post #19
epistasis wrote: post #4

Earlier quoted context omitted.

Love that thread. The top comment is excellent: > Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". > To me, that feels weird and worrying. It's like we don't yet un…

> This embodies the very fundamental difference between science and engineering. Not really though. In engineering, you have heuristics, even if you don't know why they work. In the case of deep learning / AI, there seems to be very little in the way of built up heuristic knowledge - it's just "try stuff and see what works for every problem".

I think that's also the difference between science and engineering: has the tool/technology been around enough to learn heuristics, or is everything still in the "fuck around and find out" phase?

Re: Why is machine learning 'hard'? (2016)

#22

Post by sfink »

This article annoyed me. Not because I think it overestimates any of the difficulties, but it condescendingly compares against "standard software engineers" as if their problems are so much easier and lack additional dimensions beyond algorithm and implementation. It doesn't help that they call N^4 "exponential".

Long debugging cycles are not new. In fact, the field started out that way, when computers were slow enough that even a single edit-compile-run cycle could take hours (or days and some political capital, if you go back far enough). Even today, long debugging cycles are far from being restricted to ML, especially for failures that are only seen intermittently and in production.

Performance mattering is not unique to ML. Obviously.

Data issues are not unique to ML. Anytime you need to run against a "representative workload", you'll bump into them. Heck, anytime you run against a large test suite you'll run into issues with the quality of the comparison data, especially if the tests are long in the tooth. Furthermore, anything you're doing statistics on in general is going to bump into data issues—unless doing statistics is automatically ML?

ML lacks various other dimensions of difficulty. Distributed systems. Web browsers that need to execute arbitrary code that hasn't been written yet. Backwards compatibility. Forwards compatibility. In-production databases. Cross platform development. Power usage optimization. Concurrency (ML can have this, but it doesn't always need to, and it can be in the "embarrassingly parallel" bucket). Deep dependency graphs.

The author thinks ML development is hard because that's what they work on. They are but a grasshopper.

Re: Why is machine learning 'hard'? (2016)

#23

Post by [deleted] »

sfink wrote: post #22

This article annoyed me. Not because I think it overestimates any of the difficulties, but it condescendingly compares against "standard software engineers" as if their problems are so much easier and lack additional dimensions beyond algorithm and implementation. It doesn't help that they call N^4 "exponential". Long debugging cycles are not new. In fact, the field started out that way, when computers were slow enou…

[deleted]

Re: Why is machine learning 'hard'? (2016)

#24

Post by logtempo »

epistasis wrote: post #4
dang wrote: post #3

Discussed at the time: Why is machine learning ‘hard’? - https://news.ycombinator.com/item?id=12936891 - Nov 2016 (88 comments)

Love that thread. The top comment is excellent: > Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". > To me, that feels weird and worrying. It's like we don't yet un…

> Sometimes deeper understanding provides deeper insights to the problems at hand. But often, they don't, even when the deeper understanding is beautiful. If the hammer works when you hold it a certain way, that's great, but understanding all possible ways to hold a hammer doesn't always help get the nail in better.

Is it true? I mean, in mathematics having a proof of something is way stronger than having a conjecture. And in engineering, proving that your solution is optimal is way stronger than saying "hey look, I tried many things and finally it works!".

Worse, in statistics if you throw a bunch of tests and pick the one that "works" you might have false conclusions all the time. And AI is statistics.

Sure it works to test out 10 datatset and whatever number of different machine learning, but it takes time and money and might be suboptimal from an engineering POV.

Re: Why is machine learning 'hard'? (2016)

#25

Post by Buttons840 »

ramesh31 wrote: post #11
PaulHoule wrote: post #7

The #1 thing that makes it ‘hard’ in real life is that nobody wants to make training and test sets. So we have 50,000 papers on the NIST digits but no insight into ‘would this work for a different problem?’ (Ironically the latter might have been exactly what academics would have needed to understand why these algorithms work!)

Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity.

I was just thinking the same, but I'm skeptical.

When researchers want to publish a paper, are they going to pay extra money for extra difficulty in publishing their paper? No, they'll just use whatever toy environment is free or already established and get that paper published!

Re: Why is machine learning 'hard'? (2016)

#26

Post by logiduck »

epistasis wrote: post #4
dang wrote: post #3

Discussed at the time: Why is machine learning ‘hard’? - https://news.ycombinator.com/item?id=12936891 - Nov 2016 (88 comments)

Love that thread. The top comment is excellent: > Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". > To me, that feels weird and worrying. It's like we don't yet un…

Yes, this makes it very difficult to apply ML and RL in non-simulated scenarios.

With simulated scenarios you can just replay and "sweep" across hyperparameters to find the best one.

In a realworld scenario with limited information, fine tuning hyperparameters is much harder as you quickly find yourself in local maxima.

Re: Why is machine learning 'hard'? (2016)

#27

Post by MichaelRo »

ramesh31 wrote: post #11
PaulHoule wrote: post #7

The #1 thing that makes it ‘hard’ in real life is that nobody wants to make training and test sets. So we have 50,000 papers on the NIST digits but no insight into ‘would this work for a different problem?’ (Ironically the latter might have been exactly what academics would have needed to understand why these algorithms work!)

Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity.

>> Seems like a prime startup opportunity.

Sometimes it's just ... hard. Apply some thought maybe before blindly parroting "profit!"

Reporter: "Why is it hard to cure cancer?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!"

Reporter: "Why is it hard to end World poverty?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!"

Reporter: "Why is it hard to build a warp engine?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!"

Reporter: "Why is it hard to wipe your ass using the left hand?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!"

You get the idea...

Re: Why is machine learning 'hard'? (2016)

#28

Post by beckhamc »

PaulHoule wrote: post #7

The #1 thing that makes it ‘hard’ in real life is that nobody wants to make training and test sets. So we have 50,000 papers on the NIST digits but no insight into ‘would this work for a different problem?’ (Ironically the latter might have been exactly what academics would have needed to understand why these algorithms work!)

The issue is the obsession with benchmark datasets and their flaky evaluation

Re: Why is machine learning 'hard'? (2016)

#29

Post by liuxiansheng »

ummonk wrote: post #19
epistasis wrote: post #4

Earlier quoted context omitted.

Love that thread. The top comment is excellent: > Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". > To me, that feels weird and worrying. It's like we don't yet un…

> This embodies the very fundamental difference between science and engineering. Not really though. In engineering, you have heuristics, even if you don't know why they work. In the case of deep learning / AI, there seems to be very little in the way of built up heuristic knowledge - it's just "try stuff and see what works for every problem".

I think if this was truly the case then wouldn't ML algorithm development be a solved problem with AutoML? I don't think AutoML is close to ubiquitous which means there must still be value in heuristics and a deeper understanding of our tools.

Re: Why is machine learning 'hard'? (2016)

#30

Post by graphe »

beckhamc wrote: post #28
PaulHoule wrote: post #7

The #1 thing that makes it ‘hard’ in real life is that nobody wants to make training and test sets. So we have 50,000 papers on the NIST digits but no insight into ‘would this work for a different problem?’ (Ironically the latter might have been exactly what academics would have needed to understand why these algorithms work!)

The issue is the obsession with benchmark datasets and their flaky evaluation

What else could you do to test it besides it works for me and this test said it's good at talking?
Post reply on HN