Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything is a linear model

danielroelfs.com

Post reply on HN
Print view View original

61–70 of 90 posts

Re: Everything is a linear model

#61

Post by idiotsecant »

levocardia wrote: post #53

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors ar…

Widely known in the controls engineering world for the last hundred years as 'anything is linear if you zoom in far enough'

Re: Everything is a linear model

#62

Post by whatshisface »

aeonik wrote: post #45
whatshisface wrote: post #40

Earlier quoted context omitted.

I've never actually seen a physical example of a system without a continuous first derivative. For example phase transitions, commonly touted as an example of discontinuity, don't actually occur until the matter has gone a bit over the point and a transition nucleates somewhere. The probability of a phase transition is a continuous function of temperature, with continuous derivatives. I'm skeptical that discontinuiti…

If I wanted to understand and obtain your intuition about linearity, what would you recommend?

I question how strongly I would recommend the years of working with it. :-)

Re: Everything is a linear model

#63

Post by richrichie »

whatshisface wrote: post #52
richrichie wrote: post #37

You can state and prove theorems with linear models. You can do inference and testing. This means papers and academics naturally love them. And therefore, they are everywhere. Not the case with non-linear models. We need to throw computers at them.

I think that's an overstatement. You can consider the nonlinearity as a perturbation of the linear theory, study the topology of the solutions or their symmetries, or sometimes even find exact solutions, like exist for a handful of transistor circuits.

If the model is y = a + b * x + e, then we can precisely state probability distributions of a and b given x and y for reasonably general assumptions about error e.

If the model is y = f(x) + e, where for example f is a neural net, there is not much we can say about the “quality” of parameters of f. That is, we can’t attach confidence interval. We will have to resort to expensive simulations and for most practical applications this is not workable (size of dataset).

What you say is useful in a different context. Local linearization is a very powerful idea.

Re: Everything is a linear model

#64

Post by whatshisface »

richrichie wrote: post #63
whatshisface wrote: post #52

Earlier quoted context omitted.

I think that's an overstatement. You can consider the nonlinearity as a perturbation of the linear theory, study the topology of the solutions or their symmetries, or sometimes even find exact solutions, like exist for a handful of transistor circuits.

If the model is y = a + b * x + e, then we can precisely state probability distributions of a and b given x and y for reasonably general assumptions about error e. If the model is y = f(x) + e, where for example f is a neural net, there is not much we can say about the “quality” of parameters of f. That is, we can’t attach confidence interval. We will have to resort to expensive simulations and for most practical app…

Maybe not if there are many more parameters in f than there are samples in the dataset, but if there are a handful of parameters in f then a typical step is to take the inverse of the derivative of goodness-of-fit with respect to each fit parameter, to determine the precision with which each parameter was determined by the fit.

Re: Everything is a linear model

#65

Post by vitus »

idiotsecant wrote: post #61
levocardia wrote: post #53

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors ar…

Widely known in the controls engineering world for the last hundred years as 'anything is linear if you zoom in far enough'

Well, subject to certain constraints.

You aren't guaranteed that your equilibria behave the same in the linearization of a nonlinear system if your Jacobian has any eigenvalues with real part of 0.

https://en.wikipedia.org/wiki/Hartman%E2%80%93Grobman_theore...

Re: Everything is a linear model

#66

Post by levocardia »

goosedragons wrote: post #58
carlthome wrote: post #54

Earlier quoted context omitted.

This sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs. Is there some go-to practice material I could look at? Splines I haven't touched since numerical computing exercises in school.

Simon Wood's Generalized Additive Model book.

Or, for a more approachable treatment, Semiparametric Regression with R by Harezlak, Ruppert, and Wand. A middle ground between Wood's book (which is comprehensive but can dip into math that's way over my head at times) and H/R/W is Semiparametric Regression by Ruppert, Wand, and Carroll.

I have also heard great things about Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it. His other writing is fantastic though.

Re: Everything is a linear model

#67

Post by levocardia »

carlthome wrote: post #54
levocardia wrote: post #53

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors ar…

This sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs. Is there some go-to practice material I could look at? Splines I haven't touched since numerical computing exercises in school.

If you know DNNs, you'll find the introduction of splines in Semiparametric Regression in R very intuitive - splines are introduced using what the authors call a "truncated line basis" but you already know it as the RELU function, just with a bias. Indeed, even the penalization of splines will look extremely familiar: it's basically just L2 regularization to induce smoothness.

You might also enjoy reading the "Neural Additive Model" paper from Hinton's lab, which is basically GAMs using a separate DNN as a "spline basis" for each input variable.

Re: Everything is a linear model

#68

Post by mjcohen »

levocardia wrote: post #53

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors ar…

In the 70's in CS grad school at USC, I wrote an adaptive least squares cubic spline fit routine. Kept subdividing intervals and fitting a least squares cubic spline in each interval until a criteria was met.

Fun times.

Don't have the code or writeup any more.

Re: Everything is a linear model

#69

Post by EvgeniyZh »

levocardia wrote: post #53

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors ar…

"You can make linear model non linear by adding non linearity" is kind of vacuous statement, isn't it

Re: Everything is a linear model

#70

Post by ivansavz »

EvgeniyZh wrote: post #69
levocardia wrote: post #53

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors ar…

"You can make linear model non linear by adding non linearity" is kind of vacuous statement, isn't it

No no, linear just means some expression of the form

    y = b1*x1 + b2*x2 + ... + bp*xp
which is in the heart of the model (perhaps more specific description would be linear combination). Whatever you call it, it's just some quantity y, that is constructed via an additive process from components x1...xp, and each component is multiplied by some constant (the coefficient of the linear combination).

This linear combination "core" of the model can be used directly (in which case it is geometrically a hyperplane), but you could also use non-linear transformation of the inputs or the outputs.

E.g. x1 could be a spline basis function, or log-transformed data, or anything.

Similarly, the output y can be passed though some non-linearity to force a particular output (like in logistic regression). Hence the statement is not vacuous

Post reply on HN