Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Stein's Paradox (2021)

joe-antognini.github.io

Post reply on HN
Print view View original

31–40 of 50 posts

Re: Understanding Stein's Paradox (2021)

#31

Post by mitthrowaway2 »

Sharlin wrote: post #8
mitthrowaway2 wrote: post #3

Sorry, I'm siding with the physicists here. If you're going to declare that your seemingly arbitrary choice of coordinate system is actually not arbitrary and part of your prior information about where the mean of the distribution is suspected to be, you have to put that in the initial problem statement.

You can put the origin anywhere and for almost all choices the adjustment is almost zero. But if the choice happens to be very close to the sample point, against all (prior) probabilities, then that fact affects the prior.

Not quite: If the origin is within a standard deviation of |x|² or so (depending on D), then term inside the ReLU is negative, and the adjustment is exactly zero. If the origin is moderately far away from x, then the adjustment is large. If the origin is a vast distance from x, then the adjustment is small in relative terms, but not in absolute terms. The scaling factor approaches zero for large |x| but the displacement between x and û converges toward a constant.

Either way, this is absurd unless we have some additional background information about μ other than our sample x itself. But it's easy to resolve the paradox: Since the choice of origin is arbitrary (unless it isn't!), select our coordinate system such that x = 0, then the adjustment is also zero, and then the James-Stein estimator agrees that û = x = 0.

Re: Understanding Stein's Paradox (2021)

#33

Post by rssoconnor »

zeroonetwothree wrote: post #23

I don’t understand the picture with the shaded circle. Sure the area to the left is smaller, but it also is more likely to be chosen because in a Gaussian values closer to the mean are more likely. So the picture alone doesn’t prove anything.

In the diagram the mean of the distribution is the center of the circle.

Of the set of samples a fixed distance d from the mean of the distribution, strictly less than half of them will be closer to the origin than the mean is, and strictly greater than half of them will be further from the the origin than the mean. This is true for all values of d > 0, so the result holds for all samples.

Re: Understanding Stein's Paradox (2021)

#34

Post by mitthrowaway2 »

kgwgk wrote: post #30
mitthrowaway2 wrote: post #28

Earlier quoted context omitted.

That just means that it's assuming arbitrary additional prior information about the problem, which is different than zero information.

I don't understand what you mean. Who assumes what? Take any point and shrink your least-squares estimator in that direction. You get an estimator that it's strictly better - in some technical sense - which renders the original estimator inadmissible - in some technical sense. That's a mathematical fact, it has nothing to do with prior information about the problem.

The article's presentation of the James-Stein estimator sets the arbitrary point at the origin. (My previous comments should be read in this context). Of course, we could set it anywhere, including [42,...]. Let's call it p. Regardless of where you set it, the estimator suggests that your best estimate û, of the mean μ, should be nudged a little away from x and towards p.

My point is that the choice of 'p' (or, in the article's presentation, the choice of origin) cannot truly be arbitrary because if it reduces the expected squared difference between μ and û, then it necessarily contains information about μ. If all you truly know about μ is x and σ, then you will have no way to guess in which direction you should even shift your estimate û to reduce that error.

If you do have some additional information about μ, beyond just x alone, then sure, take advantage of it! But then don't call it a paradox.

Re: Understanding Stein's Paradox (2021)

#35

Post by toth »

hyperbovine wrote: post #20

Had a bit of a chuckle at the very-2024 definition of the Stein shrinkage estimator: \hat{mu} = ReLU(…)

Ditto.

I think that ship has sailed, but I think it's unfortunate that "ReLU(x)" became a popular notation for "max(0,x)". And using the name "rectified linear unit" for basically "positive part" seems like a parody, like insisting on calling water "dihydrogen monoxide".

Re: Understanding Stein's Paradox (2021)

#36

Post by kgwgk »

mitthrowaway2 wrote: post #34
kgwgk wrote: post #30

Earlier quoted context omitted.

I don't understand what you mean. Who assumes what? Take any point and shrink your least-squares estimator in that direction. You get an estimator that it's strictly better - in some technical sense - which renders the original estimator inadmissible - in some technical sense. That's a mathematical fact, it has nothing to do with prior information about the problem.

The article's presentation of the James-Stein estimator sets the arbitrary point at the origin. (My previous comments should be read in this context). Of course, we could set it anywhere, including [42,...]. Let's call it p. Regardless of where you set it, the estimator suggests that your best estimate û, of the mean μ, should be nudged a little away from x and towards p. My point is that the choice of 'p' (or, in th…

(I cannot speak for the original article, I’ve not put the effort to fully understand it so I won’t categorically say it’s wrong but it didn’t seem right to me.)

The “paradox” is that it can truly be arbitrary! Pick a random point. Shrink your least-squares estimator. You got yourself a “better” estimator - without having any additional information.

That’s why the “Inadmissibility of the Usual Estimator for the Mean of a Multivariate Normal Distribution” paper had the impact that it had.

Re: Understanding Stein's Paradox (2021)

#37

Post by toth »

rssoconnor wrote: post #21

I think the part on "How arbitrary is the origin, really?" is not correct. The origin is arbitrary. As the Wikipedia article points you you can pick any point, whether or not it is the origin, and use the James-Stein estimator to push your estimate towards that point and it will improve one's mean squared error. If you pick a point to the left of your sample, then moving your estimate to the left will improve your me…

You make a valid point, but I feel there is something in the direction the article is gesturing at...

The mean of the n-dimensional gaussian is an element of R^n, an unbounded space. There's no uninformed prior over this space, so there is always a choice of origin implicit in some way...

As you say, you can shrink towards any point and you get a valid James-Steiner estimator that is strictly better than the naive estimator. But if you send the point you are shrinking towards to infinity you get the naive estimator again. So it feels like the fact you are implicitly selecting a finite chunk of R^n around an origin plays a role in the paradox...

Re: Understanding Stein's Paradox (2021)

#38

Post by kgwgk »

toth wrote: post #37
rssoconnor wrote: post #21

I think the part on "How arbitrary is the origin, really?" is not correct. The origin is arbitrary. As the Wikipedia article points you you can pick any point, whether or not it is the origin, and use the James-Stein estimator to push your estimate towards that point and it will improve one's mean squared error. If you pick a point to the left of your sample, then moving your estimate to the left will improve your me…

You make a valid point, but I feel there is something in the direction the article is gesturing at... The mean of the n-dimensional gaussian is an element of R^n, an unbounded space. There's no uninformed prior over this space, so there is always a choice of origin implicit in some way... As you say, you can shrink towards any point and you get a valid James-Steiner estimator that is strictly better than the naive es…

> But if you send the point you are shrinking towards to infinity you get the naive estimator again.

You get close to it but strictly speaking wouldn’t it always be better than the naive estimator?

Re: Understanding Stein's Paradox (2021)

#39

Post by mitthrowaway2 »

kgwgk wrote: post #36
mitthrowaway2 wrote: post #34

Earlier quoted context omitted.

The article's presentation of the James-Stein estimator sets the arbitrary point at the origin. (My previous comments should be read in this context). Of course, we could set it anywhere, including [42,...]. Let's call it p. Regardless of where you set it, the estimator suggests that your best estimate û, of the mean μ, should be nudged a little away from x and towards p. My point is that the choice of 'p' (or, in th…

(I cannot speak for the original article, I’ve not put the effort to fully understand it so I won’t categorically say it’s wrong but it didn’t seem right to me.) The “paradox” is that it can truly be arbitrary! Pick a random point. Shrink your least-squares estimator. You got yourself a “better” estimator - without having any additional information. That’s why the “Inadmissibility of the Usual Estimator for the Mean…

Then you'll have to clarify what you mean by "random" when you say "pick a random point".

Unless you mean that every point on a spherical surface centered on x would have a lower expected squared error than x itself?

Re: Understanding Stein's Paradox (2021)

#40

Post by toth »

kgwgk wrote: post #38
toth wrote: post #37

Earlier quoted context omitted.

You make a valid point, but I feel there is something in the direction the article is gesturing at... The mean of the n-dimensional gaussian is an element of R^n, an unbounded space. There's no uninformed prior over this space, so there is always a choice of origin implicit in some way... As you say, you can shrink towards any point and you get a valid James-Steiner estimator that is strictly better than the naive es…

> But if you send the point you are shrinking towards to infinity you get the naive estimator again. You get close to it but strictly speaking wouldn’t it always be better than the naive estimator?

Right, it's a limit at infinity
Post reply on HN