Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Stein's Paradox (2021)

joe-antognini.github.io

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Understanding Stein's Paradox (2021)

#41

Post by kgwgk »

mitthrowaway2 wrote: post #39
kgwgk wrote: post #36

Earlier quoted context omitted.

(I cannot speak for the original article, I’ve not put the effort to fully understand it so I won’t categorically say it’s wrong but it didn’t seem right to me.) The “paradox” is that it can truly be arbitrary! Pick a random point. Shrink your least-squares estimator. You got yourself a “better” estimator - without having any additional information. That’s why the “Inadmissibility of the Usual Estimator for the Mean…

Then you'll have to clarify what you mean by "random" when you say "pick a random point". Unless you mean that every point on a spherical surface centered on x would have a lower expected squared error than x itself?

We may be talking about different things.

Let's say that you have a standard multivariate normal with unknown mean mu = [a, b, c].

The usual maximum-likelihood estimator of the unknown mean when you get an observation is to take the observed value as estimate. If you observe [x, y, z] the "naive" estimator gives you the estimate mû = [x, y, z].

For any arbitrary point [p, q, r] you can define another estimator. If you observe [x, y, z] this "shrinkage" estimator gives you an estimate which is no longer precisely at [x, y, z] but is displaced in the direction of [p, q, r]. For simplicity let's say the resulting estimate is mû' = [x', y', z'].

Whatever the choice you make for [p, q, r] the "shrinkage" estimator has lower mean squared error than the "naive" estimator. The expected value of (x'-a)²+(y'-b)²+(z'-c)² is lower than the expected value of (x-a)²+(y-b)²+(z-c)².

Re: Understanding Stein's Paradox (2021)

#42

Post by rssoconnor »

toth wrote: post #37
rssoconnor wrote: post #21

I think the part on "How arbitrary is the origin, really?" is not correct. The origin is arbitrary. As the Wikipedia article points you you can pick any point, whether or not it is the origin, and use the James-Stein estimator to push your estimate towards that point and it will improve one's mean squared error. If you pick a point to the left of your sample, then moving your estimate to the left will improve your me…

You make a valid point, but I feel there is something in the direction the article is gesturing at... The mean of the n-dimensional gaussian is an element of R^n, an unbounded space. There's no uninformed prior over this space, so there is always a choice of origin implicit in some way... As you say, you can shrink towards any point and you get a valid James-Steiner estimator that is strictly better than the naive es…

> There's no uninformed prior over this space, so there is always a choice of origin implicit in some way...

You could use an uninformed improper prior.

Re: Understanding Stein's Paradox (2021)

#43

Post by kgwgk »

rssoconnor wrote: post #42
toth wrote: post #37

Earlier quoted context omitted.

You make a valid point, but I feel there is something in the direction the article is gesturing at... The mean of the n-dimensional gaussian is an element of R^n, an unbounded space. There's no uninformed prior over this space, so there is always a choice of origin implicit in some way... As you say, you can shrink towards any point and you get a valid James-Steiner estimator that is strictly better than the naive es…

> There's no uninformed prior over this space, so there is always a choice of origin implicit in some way... You could use an uninformed improper prior.

You would just need to come up with a way to pick a point at random uniformly from an unbounded space.

Re: Understanding Stein's Paradox (2021)

#44

Post by BlueTemplar »

toth wrote: post #35
hyperbovine wrote: post #20

Had a bit of a chuckle at the very-2024 definition of the Stein shrinkage estimator: \hat{mu} = ReLU(…)

Ditto. I think that ship has sailed, but I think it's unfortunate that "ReLU(x)" became a popular notation for "max(0,x)". And using the name "rectified linear unit" for basically "positive part" seems like a parody, like insisting on calling water "dihydrogen monoxide".

It hasn't "sailed" as long as they want to communicate with non-machine-learning people.

Re: Understanding Stein's Paradox (2021)

#45

Post by skinner_ »

kgwgk wrote: post #19
mitthrowaway2 wrote: post #18

Earlier quoted context omitted.

There is something magical about the origin when the result does not respect translational symmetry. In fact, in a real world setting I would probably use my first measurement to define the origin, having no other reference to reach for.

What does not respect translational symmetry? You have an estimator. If you apply shrinkage towards the origin you have another estimator. If you apply shrinkage towards [42, 42, ..., 42] you have yet another estimator. Etc. Is it a problem that different estimators produce different results?

That's my understanding as well, FWIW. This is how I would phrase it:

Shrinking helps. In R^d there's no such thing as shrinking generally, only shrinking in the direction of some point. (The point that's the fixed point of the shrinking.) Regardless of what that point is, it's a good idea to shrink.

Re: Understanding Stein's Paradox (2021)

#46

Post by rssoconnor »

kgwgk wrote: post #43
rssoconnor wrote: post #42

Earlier quoted context omitted.

> There's no uninformed prior over this space, so there is always a choice of origin implicit in some way... You could use an uninformed improper prior.

You would just need to come up with a way to pick a point at random uniformly from an unbounded space.

You can just use the function that is constantly 1 everywhere as your improper prior.

Improper priors are not distributions so they don't need to integrate to 1. You cannot sample from them. However, you can still apply Bayes' rule using improper priors and you usually get a posterior distribution that is proper.

Re: Understanding Stein's Paradox (2021)

#47

Post by kgwgk »

rssoconnor wrote: post #46
kgwgk wrote: post #43

Earlier quoted context omitted.

You would just need to come up with a way to pick a point at random uniformly from an unbounded space.

You can just use the function that is constantly 1 everywhere as your improper prior. Improper priors are not distributions so they don't need to integrate to 1. You cannot sample from them. However, you can still apply Bayes' rule using improper priors and you usually get a posterior distribution that is proper.

Sure.

The point is that you wrote that « you can pick any point […] » and when toth pointed out that « there is always a choice of origin implicit in some way » you replied that « you could use an uninformed improper prior. »

However, it seems that we agree that you cannot pick a point using an uninformed improper prior - and in any method for picking a point there will be an implicit departure from that (improper) uniform distribution.

Re: Understanding Stein's Paradox (2021)

#48

Post by rssoconnor »

kgwgk wrote: post #47
rssoconnor wrote: post #46

Earlier quoted context omitted.

You can just use the function that is constantly 1 everywhere as your improper prior. Improper priors are not distributions so they don't need to integrate to 1. You cannot sample from them. However, you can still apply Bayes' rule using improper priors and you usually get a posterior distribution that is proper.

Sure. The point is that you wrote that « you can pick any point […] » and when toth pointed out that « there is always a choice of origin implicit in some way » you replied that « you could use an uninformed improper prior. » However, it seems that we agree that you cannot pick a point using an uninformed improper prior - and in any method for picking a point there will be an implicit departure from that (improper) u…

Oh.

When I said "you can pick any point P", I meant universal quantification, i.e "for all points P", rather than a randomly chosen P.

I did say "choose P", which was pretty bad phrasing on my part.

Re: Understanding Stein's Paradox (2021)

#49

Post by m16ghost »

rssoconnor wrote: post #21

I think the part on "How arbitrary is the origin, really?" is not correct. The origin is arbitrary. As the Wikipedia article points you you can pick any point, whether or not it is the origin, and use the James-Stein estimator to push your estimate towards that point and it will improve one's mean squared error. If you pick a point to the left of your sample, then moving your estimate to the left will improve your me…

Here are some links that might help visualize what is going on:

https://www.naftaliharris.com/blog/steinviz/

https://www.youtube.com/watch?v=cUqoHQDinCM (this video actually references the original post)

My takeaway is that the volume of points which get worse as they are pulled towards point P exists in some region R. As the number of dimensions increase, region R's volume shrinks as a % of the total cloud volume, making it much more unlikely that a sample is pulled from that region. In other words, you are more likely to sample points which move closer to the center than move away, which is why the estimator is an improvement on average.

Post reply on HN