Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why squared error? (2014)

benkuhn.net

Post reply on HN
Print view View original

21–30 of 101 posts

Re: Why squared error? (2014)

#21

Post by ska »

grodeni wrote: post #14
dnautics wrote: post #12

"inner products/gaussians" - the absolute value (and also cuberoot of absolute cubes, fourth root of fourth powers) also define inner products. Likewise, there are "gaussian-like formulas" which take these powers instead of squared. However: if you look at the shape of the squareroot of sum squares, it's a circle, so you can rotate it. If you take the absolute, it's a square, so that cannot be rotated; the cuberoot o…

What kind of inner products are defined by the absolute value, cuberoot of absolute cubes, fourth root of fourth powers? I never heard of that and would be glad to learn about it.

You may find it interesting to read about Lp norms, and their relationship to inner products on vector spaces. I think the OP is mixing up norm and inner product terminology. This happens often because you derive an norm from any inner product, but the other way may not exist.

If you plot on the plane the distance = 1 line, then L_1 gives you a diamond, L_2 a circle, L_inf a square. [More precisely, the unit circle under the related metric (distance function) looks like those euclidean shapes]

Re: Why squared error? (2014)

#22

Post by klodolph »

gpsx wrote: post #17

For minimizing the square of the errors I think the good reason is because, assuming your data has gaussian probability distribution, minimizing the square error corresponds to maximizing the likelihood of the measurement, as you and others have said. Why do we assume gaussian errors? There is seldom a gaussian distribution in the real world usually because the probability for large error values doesn't not decay tha…

That's a summary of the article.

Re: Why squared error? (2014)

#23

Post by joeyo »

bitL wrote: post #16

An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack/aggregation of data we couldn't process at once in the past; these days ML + Big Data can achieve that and instead of statistics we can do computational inference instead. To me this looks like looking back to "old ways" for a reference point instead of looking forward to the unknown but more exciting.

It may not be true for all branches of machine learning (fuzzy logic, for example?), but the vast majority of modern ML techniques are equivalent to or can be viewed as types of statistical machine learning.

Re: Why squared error? (2014)

#24

Post by Normal_gaussian »

bitL wrote: post #16

An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack/aggregation of data we couldn't process at once in the past; these days ML + Big Data can achieve that and instead of statistics we can do computational inference instead. To me this looks like looking back to "old ways" for a reference point instead of looking forward to the unknown but more exciting.

ML + Big Data are a specific application of statistics

To to do anything beyond use tools other people have made (and never be sure whether results are meaningful or not) statistics are required

Of course, to make money from the ML boom you can probably get away with coincidence and correlation

Re: Why squared error? (2014)

#25

Post by Normal_gaussian »

joeyo wrote: post #23
bitL wrote: post #16

An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack/aggregation of data we couldn't process at once in the past; these days ML + Big Data can achieve that and instead of statistics we can do computational inference instead. To me this looks like looking back to "old ways" for a reference point instead of looking forward to the unknown but more exciting.

It may not be true for all branches of machine learning (fuzzy logic, for example?), but the vast majority of modern ML techniques are equivalent to or can be viewed as types of statistical machine learning.

Good point about the fuzzy logic, often the boundaries between it and statistics are... fuzzy

Re: Why squared error? (2014)

#26

Post by klodolph »

bitL wrote: post #16

An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack/aggregation of data we couldn't process at once in the past; these days ML + Big Data can achieve that and instead of statistics we can do computational inference instead. To me this looks like looking back to "old ways" for a reference point instead of looking forward to the unknown but more exciting.

Machine learning is often considered a statistical technique. The main difference seems to be that in traditional statistics, people derive practice from theory, whereas in ML people will try out techniques and figure out the theory later. That's really just a cultural difference. The techniques for analyzing ML models are all statistical to begin with.

Statistics, as a field, already used general-purpose optimization algorithms before modern ML techniques came about, so in that sense, ML just fits into an existing position in the statistical toolbox (like replacing a chisel with a 3D printer). In the other direction, statistical techniques like cross-validation are necessary for you to get your ML correct.

Re: Why squared error? (2014)

#27

Post by jostmey »

Why not KL-Divergence, which measures the error between a target distribution and the current distribution? From the perspective of Information Theory, it is the best error measurement.

Oh, and let's not forget that for a lot of problems minimizing the KL-divergence is the exact same operation as maximizing the likelihood function.

Re: Why squared error? (2014)

#29

Post by thomasahle »

throw_away_777 wrote: post #11

There is a Kaggle competition right now that uses mean absolute error, and this makes the problem substantially harder. For a practical discussion of techniques used to solve machine learning problems that use mae see the forums in: https://www.kaggle.com/c/allstate-claims-severity/forums As touched upon in the article, the objective not being differentiable is a big deal for modern machine learning methods.

> As touched upon in the article, the objective not being differentiable is a big deal for modern machine learning methods.

I'm not sure the absolute value is a big problem here. You still get a convex optimization problem. In neural networks a lot of people use ReLU or step activations functions, which are no more differentiable than the absolute value.

Re: Why squared error? (2014)

#30

Post by bitL »

klodolph wrote: post #26
bitL wrote: post #16

An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack/aggregation of data we couldn't process at once in the past; these days ML + Big Data can achieve that and instead of statistics we can do computational inference instead. To me this looks like looking back to "old ways" for a reference point instead of looking forward to the unknown but more exciting.

Machine learning is often considered a statistical technique. The main difference seems to be that in traditional statistics, people derive practice from theory, whereas in ML people will try out techniques and figure out the theory later. That's really just a cultural difference. The techniques for analyzing ML models are all statistical to begin with. Statistics, as a field, already used general-purpose optimizatio…

There is much more in ML than just statistics. I was basically asking why the "statistics filter" is so often on in ML. Neural networks don't seem a statistical technique, even if somebody uses them for regression. Yes, there is an overlap, but no, ML != statistics. As you mentioned, non-linear optimization is used in statistics on meta-level however nobody claims statistics is operations research or vice versa.
Post reply on HN