Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is machine learning 'hard'? (2016)

ai.stanford.edu

Post reply on HN
Print view View original

81–90 of 144 posts

Re: Why is machine learning 'hard'? (2016)

#81

Post by janalsncm »

p1esk wrote: post #78
janalsncm wrote: post #62

Earlier quoted context omitted.

Here’s an example of something similar. Say you have a baseline model with an AUC of 0.8. There’s a cool feature you’d like to add. After a week or two of software engineering to add it, you get it into your pipeline. AUC doesn’t budge. Is it because you added it in the wrong place? Is the feature too noisy? Is it because the feature is just a function of your existing features? Is it because your model isn’t big eno…

All of these hypotheses will take on the order of days to check. OK, but you can check them, right? How is that different from a regular software bug?

In software engineering you can test things in something on the order of seconds to minutes. Functions have fixed contracts which can be unit tested.

In ML your turnaround time is days. That alone makes things harder.

Further, some of the problems I listed are open-ended which makes it very difficult to debug them.

Re: Why is machine learning 'hard'? (2016)

#82

Post by p1esk »

wegfawefgawefg wrote: post #63
roenxi wrote: post #48

Machine learning isn't comparable to software development. It is a statistical modelling exercise. This is like asking why advertising is hard - if a non-expert wades in to a different domain then they may find it has different challenges than what they are used to! This is just a specific case of the normal things that analysts routinely deal with. The major challenges in this youthful field of machine learning are…

I dont feel like im doing statistical modeling when i do ml. Usually feels more like pipe alignment, followed by tremendous amounts of debugging.

It depends on what is it you do when you “do ml”.

Re: Why is machine learning 'hard'? (2016)

#83

Post by pbh101 »

yshvrdhn wrote: post #45

I think one of the issues is that fixing a problem is a lot harder in ML than in software engineering. You know that the model fails on this particular data point. If you have identified a bug in the code and wrote a fix did a pull request as long you are able to test the code for conditions you failed on you would have solved the problem. With modern ml especially with nueral nets as long as you don't have a way to…

I remember running into a paper from Google circa 2017 IIRC discussing the maintainability issues with machine learning models but haven’t been able to track down since. Does anyone know which one this is and have a link?

Re: Why is machine learning 'hard'? (2016)

#84

Post by hollerith »

pbh101 wrote: post #83
yshvrdhn wrote: post #45

I think one of the issues is that fixing a problem is a lot harder in ML than in software engineering. You know that the model fails on this particular data point. If you have identified a bug in the code and wrote a fix did a pull request as long you are able to test the code for conditions you failed on you would have solved the problem. With modern ml especially with nueral nets as long as you don't have a way to…

I remember running into a paper from Google circa 2017 IIRC discussing the maintainability issues with machine learning models but haven’t been able to track down since. Does anyone know which one this is and have a link?

https://research.google/pubs/machine-learning-the-high-inter...

Re: Why is machine learning 'hard'? (2016)

#85

Post by pbh101 »

hollerith wrote: post #84
pbh101 wrote: post #83

Earlier quoted context omitted.

I remember running into a paper from Google circa 2017 IIRC discussing the maintainability issues with machine learning models but haven’t been able to track down since. Does anyone know which one this is and have a link?

https://research.google/pubs/machine-learning-the-high-inter...

Looks like I found a similar one just now too from the group. Thanks!

https://proceedings.neurips.cc/paper_files/paper/2015/file/8...

Re: Why is machine learning 'hard'? (2016)

#86

Post by hcks »

janalsncm wrote: post #62
tbrownaw wrote: post #59

Earlier quoted context omitted.

> you don't often know if your data or model selection can produce the results you want. Like, not knowing if your data set actually contains anything predictive of what you're trying to predict?

Here’s an example of something similar. Say you have a baseline model with an AUC of 0.8. There’s a cool feature you’d like to add. After a week or two of software engineering to add it, you get it into your pipeline. AUC doesn’t budge. Is it because you added it in the wrong place? Is the feature too noisy? Is it because the feature is just a function of your existing features? Is it because your model isn’t big eno…

Why do you spend weeks adding something instead of directly testing all the later hypotheses?

Re: Why is machine learning 'hard'? (2016)

#87

Post by eru »

senthil_rajasek wrote: post #60
4death4 wrote: post #51

I used to work on an ML research team. In addition to what the author mentions, there is an entirely separate issue: whether or not what you're attempting to do is possible with the approach you've chosen. Consider making an iOS app. For the most part, an experienced software engineer can tell you if making a given app is possible, and they'll have a relatively clear idea about the steps required to realize the idea.…

I have always thought of ML (not DL) as phenomena that can be modelled mathematically. It turns out that not all problems have a great mathematical model like self driving cars for instance and so the search continues...

Why would self-driving cars not have a great mathematical model?

Or do you mean that the models are either black boxes (like deep learning) that we don't understand, and the white box models are not good enough?

Re: Why is machine learning 'hard'? (2016)

#88

Post by eru »

raincole wrote: post #56
4death4 wrote: post #51

I used to work on an ML research team. In addition to what the author mentions, there is an entirely separate issue: whether or not what you're attempting to do is possible with the approach you've chosen. Consider making an iOS app. For the most part, an experienced software engineer can tell you if making a given app is possible, and they'll have a relatively clear idea about the steps required to realize the idea.…

But aren't all science basically like this? If you know your hypothsis works before you do the experiments, it's not science anymore.

You are right. That's why you want to avoid doing science, when you can.

Ideally, you want to be solving engineering problems instead.

Re: Why is machine learning 'hard'? (2016)

#89

Post by eru »

p-e-w wrote: post #57
roenxi wrote: post #48

Machine learning isn't comparable to software development. It is a statistical modelling exercise. This is like asking why advertising is hard - if a non-expert wades in to a different domain then they may find it has different challenges than what they are used to! This is just a specific case of the normal things that analysts routinely deal with. The major challenges in this youthful field of machine learning are…

> Machine learning isn't comparable to software development. It is a statistical modelling exercise. It's neither of the two. Machine learning isn't comparable to any other human endeavor because in many cases, much more value comes out of the models than (seemingly) goes in. LLMs for example are punching way above their weight. The ideas underlying their software implementations are extremely simple compared to the…

I'm not sure it's as simple as you make it sound.

Lots of problems have very simple solutions. And progress often means finding simpler solutions over time.

But coming up with those solutions, and debugging them, is what's hard.

For a comparison, have a look at how pistols got simpler over the last two hundred years. Have a look at the intricate mechanism of the P08 Luger https://www.youtube.com/watch?v=9adOzT_qMq0 and compare it to a modern pistol of your choice. (And the P08 Luger is already pretty late invention.)

Or have a look at modern Ikea furniture, which can be assembled and understood by untrained members of the general public. But designing new Ikea furniture is much harder.

Re: Why is machine learning 'hard'? (2016)

#90

Post by sjwhevvvvvsj »

danielmarkbruce wrote: post #20
sjwhevvvvvsj wrote: post #13

Earlier quoted context omitted.

I do a lot of model tuning and I’m almost ashamed to say I tell GPT what performance I’m aiming for and have it generate the hyper parameters (as in just literally give me a code block). Then I see what works, tell GPT, and try again. I’m deeply uncomfortable with such a method…but my models perform quite well. Note I spend a TON of time generating the right training data, so it’s not random.

1/8th (soon to be 1/2) of the working world: "I do a lot of X and I'm almost ashamed to say I tell GPT Y then I see if it works and try again".

Well, I do want to know more about how it works. Anything important I will teach myself, it’s just hard to justify the time investment during work hours when the robot does it. Which I think is also important: these tools save time, but with downsides.
Post reply on HN