Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jensen's Inequality as an Intuition Tool (2021)

blog.moontower.ai

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Jensen's Inequality as an Intuition Tool (2021)

#11

Post by vladimirralev »

seanhunter wrote: post #10
vladimirralev wrote: post #9

Looks like a reasonable intuition guide with a couple of caveats. This intuition only works for gaussian or may be a few other distributions, not for a general p(x). Then the EV rows in the tables are not actual EV quantities, only the total is an EV, that can confuse somebody. Overall I think, the point could have been carried better with a few nice charts showing the p(x), f(p(x)) and the E.

> This intuition only works for gaussian or may be a few other distributions, not for a general p(x) I don't think that's true. It's that if X is a random variable and φ is a convex function, then φ(E[X]) ≤ E[φ(X)]. It's not necessary for X to be Gaussian, only that φ is convex. An intuitive way of thinking about it is if φ is convex then it is cup-shaped. So if I sample two points from X and draw a line φ(x_1) to φ(…

That's what I mean. The Jensen inequality applies to any distribution, but the intuition presented in the post is only good for simple distributions and all examples are gaussian/binomial-like. It would be difficult to raise the same points with something multimodal or arbitrary.

Re: Jensen's Inequality as an Intuition Tool (2021)

#12

Post by seanhunter »

vladimirralev wrote: post #11
seanhunter wrote: post #10

Earlier quoted context omitted.

> This intuition only works for gaussian or may be a few other distributions, not for a general p(x) I don't think that's true. It's that if X is a random variable and φ is a convex function, then φ(E[X]) ≤ E[φ(X)]. It's not necessary for X to be Gaussian, only that φ is convex. An intuitive way of thinking about it is if φ is convex then it is cup-shaped. So if I sample two points from X and draw a line φ(x_1) to φ(…

That's what I mean. The Jensen inequality applies to any distribution, but the intuition presented in the post is only good for simple distributions and all examples are gaussian/binomial-like. It would be difficult to raise the same points with something multimodal or arbitrary.

Aah yes

Re: Jensen's Inequality as an Intuition Tool (2021)

#14

Post by iamcreasy »

fn-mote wrote: post #3

I would have spent longer on this article but it did not define Jensen’s inequality before I got frustrated. I skimmed several section looking for it. :(

The article does it around half way through,

...the inequality the way I learned it²:

E[f(x)] ≥ f(E[x])

…if f(x) is convex

Re: Jensen's Inequality as an Intuition Tool (2021)

#15

Post by kristopolous »

daveguy wrote: post #8
fn-mote wrote: post #3

I would have spent longer on this article but it did not define Jensen’s inequality before I got frustrated. I skimmed several section looking for it. :(

There is a problem with the organization. At least one of the explanatory paragraphs in "2" of the table of contents should come before "why I find it interesting". Definitely recommend reading a little of this first: https://en.m.wikipedia.org/wiki/Jensen%27s_inequality (Could just provide a link to it near the beginning of the article for reference.)

Although I usually rail against Wikipedia math articles being gleefully esoteric jargon describing concepts in the most aggressively arcane way possible as if it's a competitive sport, the convex function article is fine. People have clearly done work on it

(People might say, well isn't it obvious what convex means. My response is no, it's dealer's choice)

Re: Jensen's Inequality as an Intuition Tool (2021)

#16

Post by vermarish »

This is really cool! I've seen Jensen's inequality used many times over in my stats/ML classes, but the traffic example here gave me an "aha" moment about how it manifests.

I like the visualizations of the expected value against the individual probabilistic components as well, though I wish there were more non-uniform distributions visualized. Perhaps if we take the traffic example and tweak the distribution to be non-uniform, that might make for a cool interactive viz.

Re: Jensen's Inequality as an Intuition Tool (2021)

#17

Post by raziel2701 »

I have no idea what Jensen's inequality means.

Goes to wikipedia

"In mathematics, Jensen's inequality, named after the Danish mathematician Johan Jensen, relates the value of a convex function of an integral to the integral of the convex function. It was proved by Jensen in 1906,[1] building on an earlier proof of the same inequality for doubly-differentiable functions by Otto Hölder in 1889.[2] Given its generality, the inequality appears in many forms depending on the context, some of which are presented below. In its simplest form the inequality states that the convex transformation of a mean is less than or equal to the mean applied after convex transformation; it is a simple corollary that the opposite is true of concave transformations"

I still have no idea what it means.

Re: Jensen's Inequality as an Intuition Tool (2021)

#18

Post by taneq »

bilater wrote: post #7
fn-mote wrote: post #3

I would have spent longer on this article but it did not define Jensen’s inequality before I got frustrated. I skimmed several section looking for it. :(

I had the exact same reaction and was about to rant here but I looked up the wikipedia article and to be fair to the author there is no simple easy way to explain this. The article actually does a decent job. :)

"If a line's getting steeper then its average height up til now is less than its average height will be later."

Re: Jensen's Inequality as an Intuition Tool (2021)

#19

Post by kxyvr »

raziel2701 wrote: post #17

I have no idea what Jensen's inequality means. Goes to wikipedia "In mathematics, Jensen's inequality, named after the Danish mathematician Johan Jensen, relates the value of a convex function of an integral to the integral of the convex function. It was proved by Jensen in 1906,[1] building on an earlier proof of the same inequality for doubly-differentiable functions by Otto Hölder in 1889.[2] Given its generality,…

A convex function is a function that is bowl shaped such a parabola, `x^2`. If you take two points and connect them with a straight line, then Jensen's inequality tells you that the function lies below this straight line. Basically, `f(cx+(1-c)y) There are a bunch of generalizations to this. It works for any convex combination of points. A convex combination of points is a weighted sum of points where the weights are positive and add to 1. If one is careful, eventually this can become an infinite convex combination of points, which means that the inequality holds with integrals.

In my opinion, the wiki article is not well written.

Re: Jensen's Inequality as an Intuition Tool (2021)

#20

Post by programjames »

raziel2701 wrote: post #17

I have no idea what Jensen's inequality means. Goes to wikipedia "In mathematics, Jensen's inequality, named after the Danish mathematician Johan Jensen, relates the value of a convex function of an integral to the integral of the convex function. It was proved by Jensen in 1906,[1] building on an earlier proof of the same inequality for doubly-differentiable functions by Otto Hölder in 1889.[2] Given its generality,…

A convex function is open to the top, like this:

  \      /
   \    /
    \  /
     \/
If you connect any two points, it lies outside the curve. Which is basically the intuition for Jensen's inequality: if you go partway between two points it's above the curve, so the weighted average of the curve at those two points is bigger than the curve at their weighted average.
Post reply on HN