Histograms for Probability Density Estimation: A Primer • Hacker News Board

Re: Histograms for Probability Density Estimation: A Primer

#11

Post by vvanirudh » Fri, Apr 12, 2024, 1:08 AM UTC

andrewla wrote: post #8

sobriquet9 wrote: post #7

Why estimate PDF through histogram then convert to CDF, when one can estimate CDF directly? Doing so also avoids having to choose bin width that can have substantial impact.

Agreed -- very odd to use a parameter (bin width) in a nonparametric estimation. Just use the raw data. In numerical analysis, broadly speaking, integrals are stable while derivatives are wild; an empirical cdf is a nice smooth integral of the messy pdf.

If sampling from the density is the only goal, then you are absolutely right. Can directly estimate empirical CDF as you pointed below. But histograms can still be useful to approximate the PDF itself? (taking the derivative of the empirical CDF to estimate PDF is wild as you said)

Re: Histograms for Probability Density Estimation: A Primer

#12

Post by abhgh » Fri, Apr 12, 2024, 3:06 AM UTC

andrewla wrote: post #8

sobriquet9 wrote: post #7

Why estimate PDF through histogram then convert to CDF, when one can estimate CDF directly? Doing so also avoids having to choose bin width that can have substantial impact.

Agreed -- very odd to use a parameter (bin width) in a nonparametric estimation. Just use the raw data. In numerical analysis, broadly speaking, integrals are stable while derivatives are wild; an empirical cdf is a nice smooth integral of the messy pdf.

"nonparametric" is somewhat of a (confusing!) misnomer in that it doesn't mean no parameters, but lots of them where the # of parameters grows with # of instances [1]. In all of these cases the models have some general parameter(s) as well.

Some simple examples would be the bin-width and bandwidth in the histogram and the kernel density estimator. A somewhat complex example would be Dirichlet Process-based Mixture Models [2]; this has a "concentration" parameter. The terminology is used outside of density estimation too, e.g., Support Vector Machines (SVM) and k-Nearest Neighbors are considered nonparametric [3].

[1] For ex, see https://stats.stackexchange.com/a/268646, or https://youtu.be/I7bgrZjoRhM?si=VOEENs773SXlEMxm&t=300

[2] https://www.gatsby.ucl.ac.uk/~ywteh/research/npbayes/dp.pdf

[3] https://stats.stackexchange.com/a/237704

Re: Histograms for Probability Density Estimation: A Primer

#13

Post by andrewla » Fri, Apr 12, 2024, 4:39 PM UTC

vvanirudh wrote: post #11

andrewla wrote: post #8

Earlier quoted context omitted.

Agreed -- very odd to use a parameter (bin width) in a nonparametric estimation. Just use the raw data. In numerical analysis, broadly speaking, integrals are stable while derivatives are wild; an empirical cdf is a nice smooth integral of the messy pdf.

If sampling from the density is the only goal, then you are absolutely right. Can directly estimate empirical CDF as you pointed below. But histograms can still be useful to approximate the PDF itself? (taking the derivative of the empirical CDF to estimate PDF is wild as you said)

Yes, if you need an approximate pdf. I've found that when I'm working non-parametrically (or in robust statistics) I like to stay in cdf space or use quantile functions more than trying to use those nasty derivatives.