11–20 of 21 posts
Yes, probability density estimation might be fun, but the simplest thing to do when comparing distributions, if you're worried about binning issues, is to plot their empirical cumulative distribution functions.
Earlier quoted context omitted.
But then you would have to choose a certain kernel and assume the data conforms to that distribution which isn't always true.
Indeed, but that estimate is likely to be less misleading in most cases than a histogram(which is just a uniform kernel that is always aligned with bin boundaries).
The smoothing give unsavvy readers a false sense of accuracy. With histogram they can at least tell it's an approximation.
Interesting paradox. I haven't seen that many statisticians using just a histogram when determining whether a certain distribution fits data reasonably. Kernel Density Estimators are a much better choice (for continuous data, like the data in the post), but they are also affected by your choice of bandwidth. When it comes down to it, like going to the doctor, sometimes the best choice is to get a second (or third!) o…
But of course, playing around with these parameters will hopefully give you a nice plot, insight into the problem and allow you to propose a proper model describing your data. Then you can fit this model to your data and extract the model parameters more precisely.
And when the distribution width of the toplogical features match your kernel sizes, of course, this PDF will look almost identical as the density plots.
Earlier quoted context omitted.
Indeed, but that estimate is likely to be less misleading in most cases than a histogram(which is just a uniform kernel that is always aligned with bin boundaries).
One particular parameter of the kernel, bandwidth, can result in highly misleading visualization given arbitrarily chosen values. Here is an example http://en.wikipedia.org/wiki/File:Comparison_of_1D_bandwidth... The smoothing give unsavvy readers a false sense of accuracy. With histogram they can at least tell it's an approximation.
http://www.umiacs.umd.edu/labs/cvl/pirl/vikas/Software/optim...
Is this basically just an effect of quantization aliasing?
There's a great story of a histogram of heights in Napoleon's army having two peaks, eliciting all sorts of theories. Reality was that height in the army was normally distributed, but that the data was collected in centimeters and people were looking at a histogram binned by inches. The middle bin contained only two centimeter counts while the bins on either side contained three each, thus having dramatically more counts.
As mentioned, one should really be using a kernel density plot instead of a histogram, except when there are already classes in the data. In R, one can simply do: library("ggplot2") library("datasets") ggplot(faithful, aes(x=eruptions)) + geom_density() + geom_rug() which gives a chart like this ( http://jean-francois.im/temp/eruptions-kde.png ). Contrast with: ggplot(faithful, aes(x=eruptions)) + geom_histogram(binw…
Is this basically just an effect of quantization aliasing?
In other words, rounding error. There's a great story of a histogram of heights in Napoleon's army having two peaks, eliciting all sorts of theories. Reality was that height in the army was normally distributed, but that the data was collected in centimeters and people were looking at a histogram binned by inches. The middle bin contained only two centimeter counts while the bins on either side contained three each,…
As mentioned, one should really be using a kernel density plot instead of a histogram, except when there are already classes in the data. In R, one can simply do: library("ggplot2") library("datasets") ggplot(faithful, aes(x=eruptions)) + geom_density() + geom_rug() which gives a chart like this ( http://jean-francois.im/temp/eruptions-kde.png ). Contrast with: ggplot(faithful, aes(x=eruptions)) + geom_histogram(binw…
Big ups on your use of GGplot--best R graphing capabilities around! In response to your update about the QQ Plot, I didn't compare against normality like you did (the article is comparing exponentials, so a normal QQ isn't the best choice). The QQ Plot just compars the quantiles of one distribution to another (could be an ecdf against a hypothesized cdf, or an ecdf against another ecdf...). Essentially, by plotting o…
And as you mention, ggplot is seriously awesome.
Interesting paradox. I haven't seen that many statisticians using just a histogram when determining whether a certain distribution fits data reasonably. Kernel Density Estimators are a much better choice (for continuous data, like the data in the post), but they are also affected by your choice of bandwidth. When it comes down to it, like going to the doctor, sometimes the best choice is to get a second (or third!) o…
Indeed, although Q-Q plots are very unlikely to be understood by people who don't have a good grasp of statistics, whereas a misleading histogram will be (and probably without knowledge of the caveats behind histograms).
To the credit of the shadier individuals in my profession, this histogram subtlety nicely highlights how it can be quite easy to bend the data to your argument using ad-hoc procedures (KDEs, hists, QQs, boxplots). A carefully chosen bin width, smoothing parameter, or covariate can present a different view of the data than some other parameter/covariate. That's why it's nice to have other statisticians capable of reproducing and disseminating the work.
