Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Model-Based Machine Learning

mbmlbook.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Model-Based Machine Learning

#21

Post by oergiR »

The "model" in the title is the model of the world, as a probabilistic model. The good thing about such a model is that it explicitly states your beliefs about the world. Once you've defined it, in theory reasoning about it is straightforward. (In practice a lot of papers get written about how to do approximate inference.) It's also straightforward to do unsupervised learning.

This is a different perspective from (most uses of) neural networks, which do not have this clear separation between the model and how to reason about it. It's funny that Chris Bishop in 1995 wrote the textbook "Neural Networks for Pattern Recognition" and now is effectively arguing against using neural networks.

You can use both by using neural networks as "factors" (the black squares) in probabilistic models.

Re: Model-Based Machine Learning

#22

Post by yummyfajitas »

xapata wrote: post #20
yummyfajitas wrote: post #18

Earlier quoted context omitted.

To me at least, the major distinction between "classical statistics" and "machine learning" is that machine learning" strives to work independently of the underlying distribution while classical statistics tries to model it. I.e., a statistician doing linear regression assumes that reality is linear (or at least differentiable) in the region of interest. A convergence proof of linear regression will use this assumpti…

> assumes that reality is linear The assumption is that a particular relationship is reasonable to model as if it were linear. No one believes reality is strictly linear. I've read your posts enough to believe you know how linear regression works. I'm criticizing your comment because it encourages a misunderstanding of traditional statistics as having nonsensical assumptions.

Out of curiosity, was my caveat "(or at least differentiable) in the region of interest" insufficient for that purpose?

I certainly didn't mean to imply that statistics has unreasonable assumptions. Merely that it tends to have stronger assumptions - and more accurate results - than machine learning. Personally I'm a huge fan of classical statistics and think it's currently underappreciated.

Re: Model-Based Machine Learning

#24

Post by xapata »

yummyfajitas wrote: post #22
xapata wrote: post #20

Earlier quoted context omitted.

> assumes that reality is linear The assumption is that a particular relationship is reasonable to model as if it were linear. No one believes reality is strictly linear. I've read your posts enough to believe you know how linear regression works. I'm criticizing your comment because it encourages a misunderstanding of traditional statistics as having nonsensical assumptions.

Out of curiosity, was my caveat "(or at least differentiable) in the region of interest" insufficient for that purpose? I certainly didn't mean to imply that statistics has unreasonable assumptions. Merely that it tends to have stronger assumptions - and more accurate results - than machine learning. Personally I'm a huge fan of classical statistics and think it's currently underappreciated.

The caveat doesn't work for a technical reason and a more important practical reason. Most relationships, even ones that aren't proper functions, can be transformed into a linear model. An absolute value function is non-differentiable for one value of the input, but it'd be perfectly fine to model with linear regression. More importantly, the audience I worry about isn't the type to pay attention to parenthetical notes using jargon. Linear is somewhat accessible jargon, but differentiable is less so. I'm not claiming that I write clearly, but I aim to write such that I don't need caveats.

Re: Model-Based Machine Learning

#25

Post by jefft255 »

yummyfajitas wrote: post #22
xapata wrote: post #20

Earlier quoted context omitted.

> assumes that reality is linear The assumption is that a particular relationship is reasonable to model as if it were linear. No one believes reality is strictly linear. I've read your posts enough to believe you know how linear regression works. I'm criticizing your comment because it encourages a misunderstanding of traditional statistics as having nonsensical assumptions.

Out of curiosity, was my caveat "(or at least differentiable) in the region of interest" insufficient for that purpose? I certainly didn't mean to imply that statistics has unreasonable assumptions. Merely that it tends to have stronger assumptions - and more accurate results - than machine learning. Personally I'm a huge fan of classical statistics and think it's currently underappreciated.

Yes it really is underappreciated. As quoted by other comments, "Most businesses think they need advanced ML and really what they need is linear regression and cleaned up data". A significant portion of businesses currently investing millions in ML should basically hire a couple of statisticians and get over it.

Re: Model-Based Machine Learning

#26

Post by xapata »

jefft255 wrote: post #25
yummyfajitas wrote: post #22

Earlier quoted context omitted.

Out of curiosity, was my caveat "(or at least differentiable) in the region of interest" insufficient for that purpose? I certainly didn't mean to imply that statistics has unreasonable assumptions. Merely that it tends to have stronger assumptions - and more accurate results - than machine learning. Personally I'm a huge fan of classical statistics and think it's currently underappreciated.

Yes it really is underappreciated. As quoted by other comments, "Most businesses think they need advanced ML and really what they need is linear regression and cleaned up data". A significant portion of businesses currently investing millions in ML should basically hire a couple of statisticians and get over it.

To be fair, the fully loaded cost of a couple statisticians (ones who can code, or combined with an engineer assistant) might be half a million or more annually.

Re: Model-Based Machine Learning

#27

Post by nl »

oergiR wrote: post #21

The "model" in the title is the model of the world, as a probabilistic model. The good thing about such a model is that it explicitly states your beliefs about the world. Once you've defined it, in theory reasoning about it is straightforward. (In practice a lot of papers get written about how to do approximate inference.) It's also straightforward to do unsupervised learning. This is a different perspective from (mo…

It's funny that Chris Bishop in 1995 wrote the textbook "Neural Networks for Pattern Recognition" and now is effectively arguing against using neural networks.

I haven't read "Neural Networks for Pattern Recognition", but his "Pattern Recognition and Machine Learning"[1] is the text for ML work including Bayesian approaches.

I don't think one should view this as "arguing against" neural networks - it's more that Bayesian approaches give you something different.

[1] http://www.springer.com/gp/book/9780387310732

Re: Model-Based Machine Learning

#28

Post by highd »

oergiR wrote: post #21

The "model" in the title is the model of the world, as a probabilistic model. The good thing about such a model is that it explicitly states your beliefs about the world. Once you've defined it, in theory reasoning about it is straightforward. (In practice a lot of papers get written about how to do approximate inference.) It's also straightforward to do unsupervised learning. This is a different perspective from (mo…

One of the most popular ways of using techniques like this is the "Variational Autoencoder". I've been working on using some alternate distributions with them as of late - it's very interesting, and quite powerful.

Re: Model-Based Machine Learning

#29

Post by nl »

highd wrote: post #28
oergiR wrote: post #21

The "model" in the title is the model of the world, as a probabilistic model. The good thing about such a model is that it explicitly states your beliefs about the world. Once you've defined it, in theory reasoning about it is straightforward. (In practice a lot of papers get written about how to do approximate inference.) It's also straightforward to do unsupervised learning. This is a different perspective from (mo…

One of the most popular ways of using techniques like this is the "Variational Autoencoder". I've been working on using some alternate distributions with them as of late - it's very interesting, and quite powerful.

How does this work? You use the VAE to model variables and then somehow get the distribution from them?

Got a link? (I know the basics of VAEs, but I'm missing how to link them to this)

Re: Model-Based Machine Learning

#30

Post by eli_gottlieb »

yummyfajitas wrote: post #18
kristjankalm wrote: post #12

Earlier quoted context omitted.

rather than downvoting i'm actually curious why you think unsupervised learning is not ML? there'd be so much less noise in these comments/discussions if we just did away with vague and illdefined labels such as ML or AI

To me at least, the major distinction between "classical statistics" and "machine learning" is that machine learning" strives to work independently of the underlying distribution while classical statistics tries to model it. I.e., a statistician doing linear regression assumes that reality is linear (or at least differentiable) in the region of interest. A convergence proof of linear regression will use this assumpti…

>A machine learning practitioner does NOT assume reality actually has a random forest out there in the world somewhere, and as a result needs to prove far more general (and less accurate) convergence results for the random forest.

Of course, most of the time nowadays, "throw a neural network or an SVM at it" doesn't really require strong convergence results... even though there are some nice analytical results for support-vector machines.

Post reply on HN