Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Tour of the Top Algorithms for Machine Learning Newbies

towardsdatascience.com

Post reply on HN
Print view View original

41–50 of 56 posts

Re: A Tour of the Top Algorithms for Machine Learning Newbies

#41

Post by claytonjy »

bunderbunder wrote: post #38
claytonjy wrote: post #26

Earlier quoted context omitted.

> generates parameters that you can easily explain to someone with no background in math I know it _seems_ that way, but there's a surprising amount of nuance there and I think we're both fooling and limiting ourselves by letting this idea fester. For one, unlike linear regression, logistic regression estimates aren't collapsible, so you can NOT interpret them as "changing this input by X changes the output by Y". Th…

I don't think that's such a big deal in practice. See http://jakewestfall.org/blog/index.php/2018/03/12/logistic-r... , for example. tl;dr: The upshot is that non-collapsibility means that I can't use LR coefficients for things that I don't really need to use them for, anyway. That doesn't feel like a crippling limitation to me. (Well, also, I have to occasionally pause to cross my fingers and say, " ceteris paribus…

I had exactly that post in mind, it really raised my awareness of these issues.

I agree with Jake's interpretation of the conditional interpretation of the estimates, but the practical issue is that virtually nobody not well-educated in statistics will do that correctly. In particular, people tend to do exactly what Jake concedes rarely makes any sense, which is comparing estimates across different model specifications.

You and I might interpret these betas just fine, but if we show them to a less stats-y audience, will they?

Re: A Tour of the Top Algorithms for Machine Learning Newbies

#42

Post by utopcell »

polkapolka wrote: post #5

The image for logistic regression is hilariously wrong. It shows the sigmoid as a decision boundary. Also dont get hung up about no free lunch theorem. That is a great result in computer science theory with little practical impact: just pick neural networks for unstructured and GBDT for structured data. For the vast majority of real-life problems (not all possible problems) these are the single best algorithms.

Exactly. I stopped reading as soon as I saw that image.

Re: A Tour of the Top Algorithms for Machine Learning Newbies

#43

Post by ummonk »

didibus wrote: post #16

I've been wondering, is nearest neighbour really ML? No part of the logic is learned from the data. It feels just like a glorified lookup table, where the look up is fuzzy to some predefined definition of nearest.

What would you think about something like local linear least squares (where it uses linear regression but locally weights the linear regression so that it is designed to fit points closer to the target)?

Is it learning patterns in the data, or is it just a glorified lookup table?

Re: A Tour of the Top Algorithms for Machine Learning Newbies

#44

Post by thanatropism »

claytonjy wrote: post #26
bunderbunder wrote: post #10

Earlier quoted context omitted.

Logistic regression isn't sexy, but it can still achieve near state-of-the-art results, is reasonably resistant to bias^H^H^H^H variance, and generates parameters that you can easily explain to someone with no background in math. There's a lot of value in all that. Especially if your deliverable is something that a business is going to use, and not just a Kaggle entry.

> generates parameters that you can easily explain to someone with no background in math I know it _seems_ that way, but there's a surprising amount of nuance there and I think we're both fooling and limiting ourselves by letting this idea fester. For one, unlike linear regression, logistic regression estimates aren't collapsible, so you can NOT interpret them as "changing this input by X changes the output by Y". Th…

It's not that difficult to compute true marginal effects from logistic regression using something like the bootstrap (if you have a distribution for your coefficients) or explicit differentiation. Every traditional stats app (Stata, SAS, etc) has this.

Re: A Tour of the Top Algorithms for Machine Learning Newbies

#45

Post by thanatropism »

jonathankoren wrote: post #29
bunderbunder wrote: post #10

Earlier quoted context omitted.

Logistic regression isn't sexy, but it can still achieve near state-of-the-art results, is reasonably resistant to bias^H^H^H^H variance, and generates parameters that you can easily explain to someone with no background in math. There's a lot of value in all that. Especially if your deliverable is something that a business is going to use, and not just a Kaggle entry.

Logistic regression is almost interpretable. It certainly looks interpretable, and it certainly good enough if you’re just trying to make PM feel better or hang some explanatory chrome in a UI, but it’s not truly interpretable. They maybe directionally interpretable, but that’s about it.

This isn't rocket science.

https://www.stata.com/support/faqs/statistics/marginal-effec...

Re: A Tour of the Top Algorithms for Machine Learning Newbies

#46

Post by ipsa »

didibus wrote: post #39
5minbreak wrote: post #20

Earlier quoted context omitted.

k-nearest neighbors classification is one of the first non-linear supervised learning algorithms. Its predictions are derived from the data sample distances. It is basically a glorified fuzzy lookup table, but then again, so can one view deep learning (fuzzy hierarchical localized lookup). Pure memorization can even outperform logistic regression, especially with big data sets, so there is some recent debate as to wh…

Interesting, though I don't totally see how deep learning would be similar. On deep learning, it is my understanding the weights are learned from the data. These are effectively constants, and represent logical rules. So in essence, the rules which relates input to output are learned from the data in deep learning. In nearest neighbour, the rule wasn't learned, we figured out the rule ourself: "use the nearest data p…

You could automatically encode a KNN model as a set of logical if-then rules: "if x1 > 10 and x2 < 3 then 4 nearest labels are [1, 1, 1, 0]" so the information is there. For KNN you could also train weights for every variable (how much should they count in the distance calculation?). For deep learning you have way more parameters and architecture choices than for nearest neighbors (mostly the distance metric and the number of neighbors to consider). After that, both learn a mapping from input data to a target.

Re: A Tour of the Top Algorithms for Machine Learning Newbies

#47

Post by jonathankoren »

thanatropism wrote: post #45
jonathankoren wrote: post #29

Earlier quoted context omitted.

Logistic regression is almost interpretable. It certainly looks interpretable, and it certainly good enough if you’re just trying to make PM feel better or hang some explanatory chrome in a UI, but it’s not truly interpretable. They maybe directionally interpretable, but that’s about it.

This isn't rocket science. https://www.stata.com/support/faqs/statistics/marginal-effec...

Knowing there is a a probabilistic relationship expressed by the coefficients and saying that “do x and y will happen” isn’t the same thing.

Re: A Tour of the Top Algorithms for Machine Learning Newbies

#48

Post by Topolomancer »

polkapolka wrote: post #5

The image for logistic regression is hilariously wrong. It shows the sigmoid as a decision boundary. Also dont get hung up about no free lunch theorem. That is a great result in computer science theory with little practical impact: just pick neural networks for unstructured and GBDT for structured data. For the vast majority of real-life problems (not all possible problems) these are the single best algorithms.

I agree that the image is wrong, but I find your suggestion a little bit too plain: _what_ kind of neural networks? How do I choose the training strategy, the learning rate, the architecture, etc.

This opens up a can of worms that can be overwhelming for beginners.

The list is not too bad actually, but the phrasing can be improved. For example, Naive Bayes should rather be introduced before LDA, as this will make LDA much more understandable.

Also, LVQ seems a little bit odd---I would rather discuss better strategies for neighbourhood enumeration (approximate kd-trees or something like that).

Re: A Tour of the Top Algorithms for Machine Learning Newbies

#49

Post by bunderbunder »

claytonjy wrote: post #41
bunderbunder wrote: post #38

Earlier quoted context omitted.

I don't think that's such a big deal in practice. See http://jakewestfall.org/blog/index.php/2018/03/12/logistic-r... , for example. tl;dr: The upshot is that non-collapsibility means that I can't use LR coefficients for things that I don't really need to use them for, anyway. That doesn't feel like a crippling limitation to me. (Well, also, I have to occasionally pause to cross my fingers and say, " ceteris paribus…

I had exactly that post in mind, it really raised my awareness of these issues. I agree with Jake's interpretation of the conditional interpretation of the estimates, but the practical issue is that virtually nobody not well-educated in statistics will do that correctly. In particular, people tend to do exactly what Jake concedes rarely makes any sense, which is comparing estimates across different model specificatio…

I guess it depends. I have the luxury of working in a very "this is machine learning, which is not to be confused with statistical inference" problem domain. It doesn't really even really make sense to interpret most the models I build as describing any sort of causal relationship, and when people are looking at the parameter estimates, they're really just trying to figure out, "What does this model think is important?"

Re: A Tour of the Top Algorithms for Machine Learning Newbies

#50

Post by thanatropism »

jonathankoren wrote: post #47
thanatropism wrote: post #45

Earlier quoted context omitted.

This isn't rocket science. https://www.stata.com/support/faqs/statistics/marginal-effec...

Knowing there is a a probabilistic relationship expressed by the coefficients and saying that “do x and y will happen” isn’t the same thing.

Logistic regression essentially gives a conditional probability function, much like linear regression gives a conditional expectation function. You can compute log odds from logistic regression -- say, conditional to all other factors being left-handed makes you twice as likely to some binary effect. People were complaining that this isn't trivially done by staring at the coefficients, but people who can't think in partial derivatives shouldn't be in this business.

OTOH: if you assume an iid framework, the probabilistic marginal effects aren't even needed to go from something like "non-bottle blondes have probability p of being haired, bottle blondes have probability q" to "painting the hair of 1000 women will generate 1000*(q-p) jobs on average". Or you can parameterize a Poisson process for rare events and report exponential/Erlang waiting times. And so on.

Post reply on HN