Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why squared error? (2014)

benkuhn.net

Post reply on HN
Print view View original

61–70 of 101 posts

Re: Why squared error? (2014)

#61

Post by Chinjut »

grodeni wrote: post #14
dnautics wrote: post #12

"inner products/gaussians" - the absolute value (and also cuberoot of absolute cubes, fourth root of fourth powers) also define inner products. Likewise, there are "gaussian-like formulas" which take these powers instead of squared. However: if you look at the shape of the squareroot of sum squares, it's a circle, so you can rotate it. If you take the absolute, it's a square, so that cannot be rotated; the cuberoot o…

What kind of inner products are defined by the absolute value, cuberoot of absolute cubes, fourth root of fourth powers? I never heard of that and would be glad to learn about it.

They don't give inner products, but they do give norms. But inner products are, in some ways, more convenient than general norms, hence squared error as opposed to other things. It's not that squared error is necessarily what you fundamentally care about; it just happens to be so conveniently analyzed, because the mathematics of inner products is convenient.

Re: Why squared error? (2014)

#62

Post by fiatjaf »

bzbarsky wrote: post #60
fiatjaf wrote: post #47

Earlier quoted context omitted.

What is a problem geometric mean solve? That was my question the entire time. When people ask "why machine learning?" the answers are "machine learning can do these things blablabla", not "you must specify the problem you're trying to solve".

> What is a problem geometric mean solve? It gives you a way to average together two things that have units that have nothing to do with each other and then compare two such averages and have the comparison make sense, as long as your units were consistent. As a silly example, say you want to average 1kg and 1m and compare that average to the average 2kg and 0.5m. With arithmetic mean, ignoring the fact that it's non…

Thank you very much. Great examples.

Re: Why squared error? (2014)

#63

Post by [deleted] »

eanzenberg wrote: post #45

Why squared error? Because you can solve the equation to minimize squared error using linear algebra in closed form. Why L2 regularization? Same reason. A closed form solution exists from linear algebra. But at the end of the day, you are most interested in the expectation value of the coefficient and minimizing the squared error gives you E[coeffs] which is the mean of the coefficients.

[deleted]

Re: Why squared error? (2014)

#64

Post by bo1024 »

eanzenberg wrote: post #45

Why squared error? Because you can solve the equation to minimize squared error using linear algebra in closed form. Why L2 regularization? Same reason. A closed form solution exists from linear algebra. But at the end of the day, you are most interested in the expectation value of the coefficient and minimizing the squared error gives you E[coeffs] which is the mean of the coefficients.

I don't think this is any more convincing than the article's reasons. There are closed forms to lots of things that aren't interesting.

Re: Why squared error? (2014)

#65

Post by im3w1l »

fiatjaf wrote: post #47
thomasahle wrote: post #31

Earlier quoted context omitted.

"Why addition?", I would ask. Different problems, different tools. You can't ask "why geometric mean?" without referring to a specific problem you're trying to solve.

What is a problem geometric mean solve? That was my question the entire time. When people ask "why machine learning?" the answers are "machine learning can do these things blablabla", not "you must specify the problem you're trying to solve".

>What is a problem geometric mean solve?

I have stock position which changed in value by a factor of 1.10 2007, by a factor of 0.80 2008 by a factor of 1.15 2009. Is there any sort of representative number x for how much the value changed per year? The final value of my stock position should come out the same when using x for every year, i.e. x * x * x = 1.10 * 0.80 * 1.15

Re: Why squared error? (2014)

#66

Post by enthdegree »

jostmey wrote: post #27

Why not KL-Divergence, which measures the error between a target distribution and the current distribution? From the perspective of Information Theory, it is the best error measurement. Oh, and let's not forget that for a lot of problems minimizing the KL-divergence is the exact same operation as maximizing the likelihood function.

kl divergence has no nice theoretical properties other than 'it is the answer to these questions'

it is also extremely poorly behaved numerically and in convergence

Re: Why squared error? (2014)

#67

Post by eanzenberg »

bo1024 wrote: post #64
eanzenberg wrote: post #45

Why squared error? Because you can solve the equation to minimize squared error using linear algebra in closed form. Why L2 regularization? Same reason. A closed form solution exists from linear algebra. But at the end of the day, you are most interested in the expectation value of the coefficient and minimizing the squared error gives you E[coeffs] which is the mean of the coefficients.

I don't think this is any more convincing than the article's reasons. There are closed forms to lots of things that aren't interesting.

I think just historically it's interesting. Every statistician was using OLS before computers because they could solve it with pen and paper, so when computers came out it was ported over. But with computers you can minimize any loss function.

However it is useful to have a closed form solution because it guarantees you actually minimized it. Other strategies to minimize functions don't guarantee that but they're still extremely useful.

Re: Why squared error? (2014)

#68

Post by nfoz »

bitL wrote: post #32
Normal_gaussian wrote: post #24

Earlier quoted context omitted.

ML + Big Data are a specific application of statistics To to do anything beyond use tools other people have made (and never be sure whether results are meaningful or not) statistics are required Of course, to make money from the ML boom you can probably get away with coincidence and correlation

Statistics means aggregate stuff and uses simplified characteristics out of semi-structured data. ML + Big Data allows you to ask precise questions like Where? How? Which ones?

As user "highd" suggested, I think you are confusing two words. I refer to Wiktionary for definitions:

Statistics: A mathematical science concerned with data collection, presentation, analysis, and interpretation.

Statistic: A quantity calculated from the data in a sample, which characterises an important aspect in the sample (such as mean or standard deviation).

If "statistics" is the term for taking a mathematical approach to understanding data, then "machine learning" is basically an applied subset of that. But you seem to be specifically using the "a statistic" definition to describe what you think the "study of statistics" is entirely concerned with.

Re: Why squared error? (2014)

#69

Post by srean »

bo1024 wrote: post #64
eanzenberg wrote: post #45

Why squared error? Because you can solve the equation to minimize squared error using linear algebra in closed form. Why L2 regularization? Same reason. A closed form solution exists from linear algebra. But at the end of the day, you are most interested in the expectation value of the coefficient and minimizing the squared error gives you E[coeffs] which is the mean of the coefficients.

I don't think this is any more convincing than the article's reasons. There are closed forms to lots of things that aren't interesting.

I cannot speak for eanzenberg but I think his comment was less about his personal justification and more about the rationalizations that have been used in the history of stats.

Gauss quite openly admitted that the choice was borne out of convenience. The justification using Normal or Gaussian distribution came later and the Gauss Markov result on conditional distribution came even later.

Even at that time when Gauss proposed the loss, it was noted by many of Gauss' peers and (perhaps by Gauss himself) that other loss functions seem more appropriate if one goes by empirical performance, in particular the L1 distance.

Now that we have the compute power to deal with L1 it has come back with a vengeance and people have been researching its properties with renewed almost earnest. In fact there is a veritable revolution that's going on right now in the ML and stats world around it.

Just as optimizing the squared loss gives you conditional expectation, minimizing the L1 error gives you conditional median. The latter is to be preferred when the distribution has a fat tail, or is corrupted by outliers. This knowledge is no where close to being new. Gauss's peers knew this.

Re: Why squared error? (2014)

#70

Post by dnautics »

grodeni wrote: post #14
dnautics wrote: post #12

"inner products/gaussians" - the absolute value (and also cuberoot of absolute cubes, fourth root of fourth powers) also define inner products. Likewise, there are "gaussian-like formulas" which take these powers instead of squared. However: if you look at the shape of the squareroot of sum squares, it's a circle, so you can rotate it. If you take the absolute, it's a square, so that cannot be rotated; the cuberoot o…

What kind of inner products are defined by the absolute value, cuberoot of absolute cubes, fourth root of fourth powers? I never heard of that and would be glad to learn about it.

hah whoops! I did confuse inner products with norms. But it is true that the L2 norm is the only one that survives transformations to arbitrary unit basis vectors.
Post reply on HN