Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything is a linear model

danielroelfs.com

Post reply on HN
Print view View original

81–90 of 90 posts

Re: Everything is a linear model

#81

Post by olivierduval »

SonOfLilit wrote: post #35
sebastianavina wrote: post #21

Earlier quoted context omitted.

"Classification of mathematical problems as linear and nonlinear is like classification of the Universe as bananas and non-bananas. " and everything turns around the same principles. For example dynamical models and PID controls. yet solving a banana, is the only thing we really know how to do. So we end up fitting everything in our banana models.

My favorite moment in university was in the first class of semester 2, where a prof said "lets look at a really small part of our thing, and assume we apply some force to it. This will make it stretch, lets assume the stretching is linear relative to the applied force". I raised my hand and asked "is this assumption supported empirically?" and he said "no, we know it's not always true, but if we don't make it we can'…

It's funny: I had the EXACT SAME PROBLEM !!!! Like "how is it science when a prof says: "this equation is too complicated too solve so let's erase these third and forth derivatives and pretend that it's the same - even if it's not - because that's all that we can do"

So I left the "physics" courses and went the "math" courses... only to learn 3 years later how to prove that this kind of approximation is mathematically sound indeed :-D

Re: Everything is a linear model

#82

Post by rcfox »

idiotsecant wrote: post #61
levocardia wrote: post #53

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors ar…

Widely known in the controls engineering world for the last hundred years as 'anything is linear if you zoom in far enough'

Alternatively, anything is linear if you add enough dimensions.

Re: Everything is a linear model

#83

Post by AstralStorm »

olivierduval wrote: post #81
SonOfLilit wrote: post #35

Earlier quoted context omitted.

My favorite moment in university was in the first class of semester 2, where a prof said "lets look at a really small part of our thing, and assume we apply some force to it. This will make it stretch, lets assume the stretching is linear relative to the applied force". I raised my hand and asked "is this assumption supported empirically?" and he said "no, we know it's not always true, but if we don't make it we can'…

It's funny: I had the EXACT SAME PROBLEM !!!! Like "how is it science when a prof says: "this equation is too complicated too solve so let's erase these third and forth derivatives and pretend that it's the same - even if it's not - because that's all that we can do" So I left the "physics" courses and went the "math" courses... only to learn 3 years later how to prove that this kind of approximation is mathematicall…

Should have taken one more course about chaos theory which is the special case when this approach fails.

Re: Everything is a linear model

#84

Post by disgruntledphd2 »

levocardia wrote: post #66
goosedragons wrote: post #58

Earlier quoted context omitted.

Simon Wood's Generalized Additive Model book.

Or, for a more approachable treatment, Semiparametric Regression with R by Harezlak, Ruppert, and Wand. A middle ground between Wood's book (which is comprehensive but can dip into math that's way over my head at times) and H/R/W is Semiparametric Regression by Ruppert, Wand, and Carroll. I have also heard great things about Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (stil…

> Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it.

Very very good indeed, with the exception that he basically ignores compute time and efficiency. I learned a bunch, but applying his approach to the kinds of datasets I deal with (much larger and with pretty strict compute budgets) was very difficult.

Re: Everything is a linear model

#85

Post by fho »

levocardia wrote: post #53

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors ar…

Can you recommend some resources to go from step 3 to 4?

So far I have been successfully using GLMMs where appropriate, but then jumped to implementing completely arbitrary models by fitting them to the data (plus bootstrapping).

But that takes ages for most problems as you say

Re: Everything is a linear model

#86

Post by aeonik »

whatshisface wrote: post #62
aeonik wrote: post #45

Earlier quoted context omitted.

If I wanted to understand and obtain your intuition about linearity, what would you recommend?

I question how strongly I would recommend the years of working with it. :-)

Like pure math? Engineering?

Study it in school?

There are a lot of linear problems that I'm interested in.

Do you use a computer algebra system?

Do you have a few books?

Re: Everything is a linear model

#87

Post by ronald_raygun »

blitzar wrote: post #80
kqr wrote: post #77

This is why I feel like I couldn't become a credible AI/ML consultant. I would just throw everything into a linear model, make some progress, and call it a day.

99% of AI / ML consultants are just throwing everything into a linear model, make no progress, slap a big AI sticker on the box and call it a day. You would have fitted right in.

Wasn’t there an OpenAI paper where they showed that MM multiplication with the numerical imprecision of floats was enough to get general function learning?

You might not even need anything else…

Re: Everything is a linear model

#88

Post by blitzar »

ronald_raygun wrote: post #87
blitzar wrote: post #80

Earlier quoted context omitted.

99% of AI / ML consultants are just throwing everything into a linear model, make no progress, slap a big AI sticker on the box and call it a day. You would have fitted right in.

Wasn’t there an OpenAI paper where they showed that MM multiplication with the numerical imprecision of floats was enough to get general function learning? You might not even need anything else…

One almost certainly does not need anything else ... unless what you want is big piles of investor cash, a Scrooge McDuck swimming pool quantity of investor cash - then you need to call whatever maths you do AI.

Re: Everything is a linear model

#89

Post by fnordpiglet »

whatshisface wrote: post #60
fnordpiglet wrote: post #59

Earlier quoted context omitted.

Quantized space is absolutely discontinuous, and tunneling is a discontinuous system. In fact assuming the universe is quantum it’s discontinuous in reality but the appearance is continuous. But these distinctions aren’t super useful unless you’re dealing with these sorts of effects. Continuity is the approximation, discontinuity is the reality. But depending on what’s useful we use the mathematics that help us.

Tunneling currents are continuous in every parameter, although I admit that when you're dealing with particles you have continuous probability distributions with continuously varying means, rather than continua of matter. (But that should count, because all macroscopic variables are expectation values.)

Specifically at quantized space and time levels everything is discrete even distribution functions. There’s no sense in having a continuous spacial distribution sub Planck lengths.

Re: Everything is a linear model

#90

Post by epgui »

hackerlight wrote: post #16
iamcreasy wrote: post #6

Earlier quoted context omitted.

I presume you are implying that linear model only mandates linear relationship between predictor and regression coefficients?

Linear models are a linear combination of possibly non-linear regressors. The linearity is strictly in the parameters, not in whatever you're adding up. A neural network can be pedantically referred to as a linear model of the form y = a + b*neural_network, for example. Here, y is a linear model (even though neural_network isn't).

IMO it's not very pedantic... It's pretty much exactly what it is! (I'm not too sure about the example equation you give, however)
Post reply on HN