Earlier quoted context omitted.
While I get your point, it doesn't carry too much weight, because you can (and we often read this) claim the opposite: Linear regression, for all its faults, forces you to be very selective about parameters that you believe to be meaningful, and offers trivial tools to validate the fit (i.e. even residuals, or posterior predictive simulations if you want to be fancy). ML and beyond, on the other hand, throws you in a…
That particular xkcd was funny until the LLMs came around
How linear regression works intuitively and how it leads to gradient descent
11–20 of 107 posts
Re: How linear regression works intuitively and how it leads to gradient descent
#12Earlier quoted context omitted.
That particular xkcd was funny until the LLMs came around
Aren't LLMs also a pile of linear algebra?
This can be seen as another occurence of the "bitter lesson": http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Re: How linear regression works intuitively and how it leads to gradient descent
#13Re: How linear regression works intuitively and how it leads to gradient descent
#14One interesting property of least squares regression is that the predictions are the conditional expectation (mean) of the target variable given the right-hand-side variables. So in the OP example, we're predicting the average price of houses of a given size. The notion of predicting the mean can be extended to other properties of the conditional distribution of the target variable, such as the median or other quanti…
Re: How linear regression works intuitively and how it leads to gradient descent
#15There are plenty of error formulations that give a smooth loss function, and many even a convex one, but most don't have analytical solutions so they are solved via numerical optimization like GD.
The main message is IMHO correct though: square error (and its implicit gaussian noise assumption) is all too often used just per convenience and tradition.
Re: How linear regression works intuitively and how it leads to gradient descent
#16This is very light and approachable but stops short of building the statistical intuition you want here. They fixate on the smoothness of squared errors without connecting that to the gaussian noise model and establishing how that relates to the predictive power against natural sorts of data.
Re: How linear regression works intuitively and how it leads to gradient descent
#17Earlier quoted context omitted.
Aren't LLMs also a pile of linear algebra?
That's the point, yes. "Piling up more and more data then stirring it until it works" stopped being a joke and turned out to be a practical approach. This can be seen as another occurence of the "bitter lesson": http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Re: How linear regression works intuitively and how it leads to gradient descent
#18Re: How linear regression works intuitively and how it leads to gradient descent
#19Earlier quoted context omitted.
Aren't LLMs also a pile of linear algebra?
That's the point, yes. "Piling up more and more data then stirring it until it works" stopped being a joke and turned out to be a practical approach. This can be seen as another occurence of the "bitter lesson": http://www.incompleteideas.net/IncIdeas/BitterLesson.html
I indeed find the lesson that it describes unbearably bitter. Searching and learning, as used by the article, may discover patterns and results (due to infinite scaling of computation) that we, humans, are physically uncapable of discovering -- however, all those learnings will have no meaning, they will not expose any causality. This is what I find unbearable, as it implies that the real world must ultimately remain impervious to human cognizance; it implies that our meaning- and causality-based human reasoning ultimately falls short to model the world, while general, computation-only methods (given ever-growing computing power) at least "converges" to a faithful (but meaningless) description of the world.
See examples like protein folding, medicine research, AI-assisted diagnosis, self driving cars. We're going to rely on their results, but we'll never know why those results work. We're not going to reject self-driving cars if those cars save lives per same distance driven and/or same time driven; however, we're going to sit in, and drive, those cars blind. To me, that's an unbearable thought, even apart from the possibility that at some point the system might break down, and cause a huge accident inexplicably. An inexplicable misbehavior of the system is of course catastrophic, but to me, even the inexplicable proper behavior of the system is an unsettling thought -- because it is inexplicable.
Edited to add: I think the phrase "how we think we think" is awesome in the essay. We don't even know how our reasoning works, so trying to "machinize" those misconceptions is likely bound to fail.
Re: How linear regression works intuitively and how it leads to gradient descent
#20Speaking of linear regression, can any of you recommend an online course or book that deep dives into fitting linear models?
I did the Stats I -> II -> II pipeline at uni but you should be fitting basic linear models by the end of Stats I