Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is machine learning 'hard'? (2016)

ai.stanford.edu

Post reply on HN
Print view View original

131–140 of 144 posts

Re: Why is machine learning 'hard'? (2016)

#131

Post by PaulHoule »

Mltrw wrote: post #39
whiplash451 wrote: post #12

Earlier quoted context omitted.

You’re not paying tribute to MNIST-1D and many other datasets (including the massive segmentation dataset released by Meta with SAM). Read the literature before lecturing the community.

We still don't have enough data and people are still wasting their time with trying to extend algorithms instead of making better training data. I've worked on a dozen ml projects two of them before Alex net came out and I've never gone wrong by spending 80% of my time creating a dataset specific to the problem and then using whatever algorithm is top dog right now. Labelled data is king.

Personally I am happy to use a model that isn't quite "top dog".

I have a classification task where I can train multiple models to do automated evaluation in about 3 minutes using BERT + classical ML. The models are consistently good.

Sometimes you can do better fine-tuning the BERT model with your training set but a single round takes 30 minutes. The best fine-tuned models are about as good as my classical ML models but the results are not consistent and I haven't developed a reliable training procedure and if I did it would probably take 3 hours or more because I'd have to train multiple models with different parameters.

Even if I could get 82% AUC over 81% AUC I'm not so sure it is worth the trouble, and if I really felt I needed a better AUC (the number I live by, not the usually useless 'accuracy' and F1) I could develop a stacked model based on my simple classifier which shouldn't be too hard because of the rapid cycle time it makes possible.

My favorite arXiv papers are not the ones where people develop "cutting edge" methods but where people have a run-of-the-mill problem and apply a variety of run-of-the-mill methods. I find people like that frequently get results like mine so they're quite helpful.

Re: Why is machine learning 'hard'? (2016)

#132

Post by p1esk »

janalsncm wrote: post #81
p1esk wrote: post #78

Earlier quoted context omitted.

All of these hypotheses will take on the order of days to check. OK, but you can check them, right? How is that different from a regular software bug?

In software engineering you can test things in something on the order of seconds to minutes. Functions have fixed contracts which can be unit tested. In ML your turnaround time is days. That alone makes things harder. Further, some of the problems I listed are open-ended which makes it very difficult to debug them.

I've been an ML researcher for the last 11 years. Last week I spent 3 days debugging an issue in my code which had nothing to do with ML. It was my search algorithm not properly modifying the state of an object in my quantization algorithm. Individually, both algorithms worked correctly but the overall result was incorrect.

Looking back at my career, the hardest bugs to spot were not in ML, but in distributed systems, parallel data processing algorithms, task scheduling, network routing, and in dynamic programming. Of course I also had a ton of ML related bugs, but over the years I developed tools and intuition to deal with them, so usually I can narrow down an issue (like impacted accuracy, or training not converging) fairly quickly. I don't think these kind of bugs are fundamentally different from any other bugs in software engineering. You typically try to isolate an issue by simplifying the environment, break it down into parts, test on toy problems, trace program execution and print out or visualize values.

Re: Why is machine learning 'hard'? (2016)

#133

Post by p1esk »

blackbear_ wrote: post #120
p1esk wrote: post #78

Earlier quoted context omitted.

All of these hypotheses will take on the order of days to check. OK, but you can check them, right? How is that different from a regular software bug?

What makes some bugs in ML algorithms hard to spot is that many of then hinder, but do not prevent, the model from learning. They can be really hard to spot because you do see the model learning and getting better, and yet without that bug the predictions could be even more accurate. Only with domain experience you can tell that something might be wrong. Moreover, this kind of issues are usually related to the mathem…

Only with domain experience you can tell that something might be wrong.

Obviously. How is this different from any other field of science or engineering?

you need to understand the theoretical motivation of things and check all operations one by one.

Again, this is true when debugging any complex system. How else would you debug it?

a bug there where we were normalizing on the wrong dimension of a tensor

If you describe the methodology you used to debug it, it will probably be applicable to debugging a complicated issue in any other SWE domain.

Re: Why is machine learning 'hard'? (2016)

#134

Post by mvkel »

This is a dumb question: could we not make an ML debugger out of ML?

It seems ML, or at least model training and fine tuning, is all about pattern recognition.

Could the same thing not be done where a model is trained on ML algos running correctly vs. not and it performs the pattern recognition (i.e. intuition) to find the root cause?

Re: Why is machine learning 'hard'? (2016)

#135

Post by janalsncm »

DeathArrow wrote: post #104
janalsncm wrote: post #62

Earlier quoted context omitted.

Here’s an example of something similar. Say you have a baseline model with an AUC of 0.8. There’s a cool feature you’d like to add. After a week or two of software engineering to add it, you get it into your pipeline. AUC doesn’t budge. Is it because you added it in the wrong place? Is the feature too noisy? Is it because the feature is just a function of your existing features? Is it because your model isn’t big eno…

>AUC doesn’t budge. Is it because you added it in the wrong place? Is the feature too noisy? Is it because the feature is just a function of your existing features? Is it because your model isn’t big enough to learn the new feature? Is there a logical bug in your implementation? Or is it because lack of expertise and experience and because someone tries stuff blindly without understanding a bit in the hope they will…

You could say that. No one has even a decade of experience with transformers. Most of this stuff is pretty new.

More broadly though, it’s because there aren’t great first principles reasons for why things work or not.

Re: Why is machine learning 'hard'? (2016)

#136

Post by janalsncm »

p1esk wrote: post #133
blackbear_ wrote: post #120

Earlier quoted context omitted.

What makes some bugs in ML algorithms hard to spot is that many of then hinder, but do not prevent, the model from learning. They can be really hard to spot because you do see the model learning and getting better, and yet without that bug the predictions could be even more accurate. Only with domain experience you can tell that something might be wrong. Moreover, this kind of issues are usually related to the mathem…

Only with domain experience you can tell that something might be wrong. Obviously. How is this different from any other field of science or engineering? you need to understand the theoretical motivation of things and check all operations one by one. Again, this is true when debugging any complex system. How else would you debug it? a bug there where we were normalizing on the wrong dimension of a tensor If you descri…

Because the difference is that statistical models are by definition somewhat stochastic. Some incorrect answers are to be expected, even if you do everything right.

In software engineering you have test code. 100% of your tests should pass. If one doesn’t you can debug it until it does.

Re: Why is machine learning 'hard'? (2016)

#137

Post by blackbear_ »

p1esk wrote: post #133
blackbear_ wrote: post #120

Earlier quoted context omitted.

What makes some bugs in ML algorithms hard to spot is that many of then hinder, but do not prevent, the model from learning. They can be really hard to spot because you do see the model learning and getting better, and yet without that bug the predictions could be even more accurate. Only with domain experience you can tell that something might be wrong. Moreover, this kind of issues are usually related to the mathem…

Only with domain experience you can tell that something might be wrong. Obviously. How is this different from any other field of science or engineering? you need to understand the theoretical motivation of things and check all operations one by one. Again, this is true when debugging any complex system. How else would you debug it? a bug there where we were normalizing on the wrong dimension of a tensor If you descri…

> How is this different from any other field of science or engineering?

The difference is that in most cases it is not so clear how well any given approach will work in a given scenario. Often the only option is to try, and if performance is not satisfying it is not easy to find a reason for it. Besides bugs or wrong model choice, it could be wrong training parameters, the quality or quantity of the data, and who knows how much more you would need.

It's not necessarily different from SWE, problem solving is a general skill, the difficulty comes from the fact that there is no clear definition of "it works" and that there are no guidelines or templates to follow to find out what is wrong, if anything at all. In particular, many issues are not about the code.

Re: Why is machine learning 'hard'? (2016)

#138

Post by Sohcahtoa82 »

4death4 wrote: post #51

I used to work on an ML research team. In addition to what the author mentions, there is an entirely separate issue: whether or not what you're attempting to do is possible with the approach you've chosen. Consider making an iOS app. For the most part, an experienced software engineer can tell you if making a given app is possible, and they'll have a relatively clear idea about the steps required to realize the idea.…

> In addition to what the author mentions, there is an entirely separate issue: whether or not what you're attempting to do is possible with the approach you've chosen.

I had a fun project I tried once. I wanted to see if a neural network could be fed a Bitcoin public key and output the private key. To make things simple, I tried to see if it could even predict a single bit of the private key. 256 bits of input, 1 bit of output.

I created a set of 1000 public/private key pairs to act as the test set. Then, I looped, generating a new set of 1000 key pairs, trained for several epochs, then tested on the test set.

After 3 days of training (granted, on a CPU several years ago), the results on the test set did not converge. Nearly every trial on the test set was 47-53% correct. I think I had one run that was 60% correct, but that was likely pure lock. Do enough trials of 1000 coin flips and you'll likely find one where you get at least 600 heads.

Back to your original comment...Is what I was attempting to do even possible? Did my network need more layers? More nodes per layer? Or was it simply not possible?

Based on what I know about cryptography, it shouldn't be possible. A friend of mine said that for a basic feed-forward neural network to solve a problem, the problem has to be solvable via a massive polynomial that the training will suss out. Hashing does not have such a polynomial, or it would indicate the algorithm is broken.

But I still always wonder...what if I had a bigger network...

Re: Why is machine learning 'hard'? (2016)

#139

Post by coldtea »

throwaway8877 wrote: post #58
owlninja wrote: post #54

Earlier quoted context omitted.

You nailed where I currently stand. At my company I've been a jack of all trades but mostly software/dba work. My boss and I were very excited about ML when the hype cycle was taking off several years ago and completed a successful project. Fast forward to today, I got loaned out to another team that lost their data scientist, and for the first time in my career I'm having to say - "I don't think we can do what you w…

The situation is not bad then. Can they collect more data? Can they generate more data?

A more relevant question would be:

Is "not enough data" their problem, or the kind of data?

Re: Why is machine learning 'hard'? (2016)

#140

Post by steve_adams_86 »

owlninja wrote: post #54
4death4 wrote: post #51

I used to work on an ML research team. In addition to what the author mentions, there is an entirely separate issue: whether or not what you're attempting to do is possible with the approach you've chosen. Consider making an iOS app. For the most part, an experienced software engineer can tell you if making a given app is possible, and they'll have a relatively clear idea about the steps required to realize the idea.…

You nailed where I currently stand. At my company I've been a jack of all trades but mostly software/dba work. My boss and I were very excited about ML when the hype cycle was taking off several years ago and completed a successful project. Fast forward to today, I got loaned out to another team that lost their data scientist, and for the first time in my career I'm having to say - "I don't think we can do what you w…

> they just don't have enough useful data...

I'm not well-versed in this, or not as well as you are, but this has been my conclusion as well about a lot of ML project ideas from teams I've been on.

You need so much data to do useful things. Especially the magical kinds of things people tend to want to do. I think these types of datasets are on a scale most software developers typically don't see. Even with the data in hand, it's nothing like trivial to determine how to do something half-way useful with it.

Post reply on HN