Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything is a linear model

danielroelfs.com

Post reply on HN
Print view View original

51–60 of 90 posts

Re: Everything is a linear model

#51

Post by whatshisface »

blt wrote: post #44
whatshisface wrote: post #40

Earlier quoted context omitted.

I've never actually seen a physical example of a system without a continuous first derivative. For example phase transitions, commonly touted as an example of discontinuity, don't actually occur until the matter has gone a bit over the point and a transition nucleates somewhere. The probability of a phase transition is a continuous function of temperature, with continuous derivatives. I'm skeptical that discontinuiti…

Any physical system that makes/breaks contact, such as walking robots. Sure, the foot is not perfectly rigid and technically is a stiff spring. But from a computational perspective, problems still bear all the hallmarks of a discontinuous system such as requiring a very short integration step.

Yes, that's another example of the discontinuity existing in an idealized model, but not real life.

Re: Everything is a linear model

#52

Post by whatshisface »

richrichie wrote: post #37

You can state and prove theorems with linear models. You can do inference and testing. This means papers and academics naturally love them. And therefore, they are everywhere. Not the case with non-linear models. We need to throw computers at them.

I think that's an overstatement. You can consider the nonlinearity as a perturbation of the linear theory, study the topology of the solutions or their symmetries, or sometimes even find exact solutions, like exist for a handful of transistor circuits.

Re: Everything is a linear model

#53

Post by levocardia »

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML).

My series of lm/glm/gam/gamm revelations was:

1. All t-tests and ANOVA flavors are just linear models

2. Linear models are just a special case of generalized linear models (GLMs), which can deal with binary or count data too

3. All linear models and GLMs are just special cases of generalized linear mixed models, which can deal with repeated measures, grouped data, and other non-iid clustering

4. Linearity is usually a bad assumption, which can easily be overcome via splines

5. Estimating a spline smoothing penalty is the same thing as estimating the variance for a random effect in a a mixed model, so #3 and #4 can be combined for free

And then you end up with a generalized additive mixed model (GAMM), which can model smooth nonlinear functions of many variables, smooth interaction surfaces between variables (e.g. latitude/longitude), handle repeated measurements and grouping, and deals with many types of outcomes, including continuous, binary yes/no, count, ordinal categories, or survival time measurements.

All while yielding statistically valid confidence intervals, and typically only taking a few minutes of CPU time even on datasets with hundreds of thousands / millions of datapoints.

Re: Everything is a linear model

#54

Post by carlthome »

levocardia wrote: post #53

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors ar…

This sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs.

Is there some go-to practice material I could look at?

Splines I haven't touched since numerical computing exercises in school.

Re: Everything is a linear model

#55

Post by [deleted] »

levocardia wrote: post #53

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors ar…

[deleted]

Re: Everything is a linear model

#56

Post by gwern »

Horffupolde wrote: post #39
matteoraso wrote: post #34

A cool thing about linear models is that they can be used to model non-linear correlations by using transformations. For example, an exponential function can be made linear if you just take the logarithm.

Not all functions have a trivial transformation.

Doesn't have to be 'trivial'. It could be a trillion-parameter MoE LLM spitting out 10,000-large embeddings. But a lot of that will act linearly and can, in theory, be put into your linear model easily & profitably.

Re: Everything is a linear model

#58

Post by goosedragons »

carlthome wrote: post #54
levocardia wrote: post #53

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors ar…

This sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs. Is there some go-to practice material I could look at? Splines I haven't touched since numerical computing exercises in school.

Simon Wood's Generalized Additive Model book.

Re: Everything is a linear model

#59

Post by fnordpiglet »

whatshisface wrote: post #51
blt wrote: post #44

Earlier quoted context omitted.

Any physical system that makes/breaks contact, such as walking robots. Sure, the foot is not perfectly rigid and technically is a stiff spring. But from a computational perspective, problems still bear all the hallmarks of a discontinuous system such as requiring a very short integration step.

Yes, that's another example of the discontinuity existing in an idealized model, but not real life.

Quantized space is absolutely discontinuous, and tunneling is a discontinuous system. In fact assuming the universe is quantum it’s discontinuous in reality but the appearance is continuous. But these distinctions aren’t super useful unless you’re dealing with these sorts of effects. Continuity is the approximation, discontinuity is the reality. But depending on what’s useful we use the mathematics that help us.

Re: Everything is a linear model

#60

Post by whatshisface »

fnordpiglet wrote: post #59
whatshisface wrote: post #51

Earlier quoted context omitted.

Yes, that's another example of the discontinuity existing in an idealized model, but not real life.

Quantized space is absolutely discontinuous, and tunneling is a discontinuous system. In fact assuming the universe is quantum it’s discontinuous in reality but the appearance is continuous. But these distinctions aren’t super useful unless you’re dealing with these sorts of effects. Continuity is the approximation, discontinuity is the reality. But depending on what’s useful we use the mathematics that help us.

Tunneling currents are continuous in every parameter, although I admit that when you're dealing with particles you have continuous probability distributions with continuously varying means, rather than continua of matter. (But that should count, because all macroscopic variables are expectation values.)
Post reply on HN