Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I've stopped using box plots (2021)

nightingaledvs.com

Post reply on HN
Print view View original

221–230 of 258 posts

Re: I've stopped using box plots (2021)

#221

Post by blueflow »

crazygringo wrote: post #191
cjk2 wrote: post #19

Earlier quoted context omitted.

This is also wrong. Gaussian curves are symmetric. Box plots do not have to be. In fact representing skew in a batch is one of the fundamental purposes of them.

But representing skew is precisely to show how "off" from a Guassian it is. Because real data is never perfectly Guassian, or perfectly anything. But the idea of a box plot is that it's for data which is in theory Gaussian or a similar unimodal kind of bell-shaped curve. Then you can look at the box plot and see if it actually is -- are the two boxes roughly equal-sized? Are the lines a bit longer than the boxes but…

You model skew in Gaussian distributions by adding an exponential parameter.

Re: I've stopped using box plots (2021)

#222

Post by blueflow »

ColFrancis wrote: post #199
blueflow wrote: post #128

Earlier quoted context omitted.

Look at this SVG from wikipedia: https://upload.wikimedia.org/wikipedia/commons/1/1a/Boxplot_... When you calculate the box plot using normal distribution parameters, the outliers are outside the outer bracket. If you split the dataset into 4 equal parts, the bracket will be larger because the outliers are still inside it. The methodologies are not equal. This thread is the first time i heard people do the "split dat…

For what it's worth, you've convinced me that my beloved box plots need to be explained if I want to use them again. The SVG you've provided clearly shows that the box plot splits the data in 4. The interquartile range (IQR) is clearly marked and it even has a comparison for what the standard deviation (variance) measure would be. Secondly, if the data truly came from a normal distribution, there are no outliers. Out…

When i said "splitting", i meant it like my parent explained: Basically sorting your datasets and then splitting into quarters.

What you want to explain to me (IMHO to the wrong person) is the correct approach of calculating a mean and standard deviation and drawing the box from that. Lets stay with that (and thats what i said earlier in the thread)

After i wrote the post you replied to, i realized that the pure "splitting" method for box plots is nonsensical since the outer brackets interval is determined by the two most extreme values. They are too random to be meaningful. It does not make sense to draw a box plot from that.

Re: I've stopped using box plots (2021)

#223

Post by hoseja »

seanhunter wrote: post #212
sanderjd wrote: post #174

Earlier quoted context omitted.

Could you summarize the criticisms in this (pretty long) video, and what she is proposing as a better alternative (beanplots? or is she criticizing those too?)? I couldn't figure it out from perusing the transcript. I think it's useful to be able to compare the approximate shapes of histograms during exploratory data analysis. Is the thesis of this criticism that this isn't actually a useful thing to do, or that viol…

The summary is she is saying you almost always want to show one of two things (and not both): 1) To show the distribution, in which case just the histogram arranged horizontally in the traditional fashion is far better than a violin plot with 2 copies of the histogram vertically and some extra quartile stuff tacked on, especially since lots of standard libraries to do violin plots do kde with very extreme smoothing s…

3) They look like THAT

Re: I've stopped using box plots (2021)

#224

Post by blueflow »

crazygringo wrote: post #190
Evidlo wrote: post #83

Earlier quoted context omitted.

> So you can no longer reverse the process I've never understood this to be the purpose of a boxplot, only a means of visualizing a distribution's quartiles. You've gotten a flood of comments from upset people, so I'll keep it short by saying that a boxplot doesn't actually do what you claim for Gaussians, as the 0 and 100 percentile "whiskers" would be at plus/minus infinity. As for a bounded bell-shaped distributio…

> as the 0 and 100 percentile "whiskers" would be at plus/minus infinity The point is not to plot an ideal Gaussian, the point is to plot the data. In real life the whiskers are the actual minimum and maximum values observed.

> In real life the whiskers are the actual minimum and maximum values observed.

Look at this: https://upload.wikimedia.org/wikipedia/commons/1/1a/Boxplot_...

0.7% of all values are outside the whiskers.

Re: I've stopped using box plots (2021)

#225

Post by s1artibartfast »

callalex wrote: post #215
kaitai wrote: post #171

Earlier quoted context omitted.

I deal with a lot of business people who have processes that rely on 15th/85th percentile, or 25th/75th percentile. They want to see the median, the low/high percentiles, the max/min or outliers, and they don't want to see all the data points jittered in between. It's just overwhelming extraneous information. They in fact like tables with those numbers written down, but they want to compare ten different (time series…

Who drew those boundaries at 15/85? What makes those boundaries useful or correct?

It sounds like they are business relevant parameters. They are self selected and independent of the data or distribution.

The point is that they are parameters of relevance to observer.

I work in medicine sometimes work with box-plots for this reason. The questions "what is the 25th percentile outcome" is perfectly legitimate

Re: I've stopped using box plots (2021)

#226

Post by thaumasiotes »

blueflow wrote: post #217
thaumasiotes wrote: post #203

Earlier quoted context omitted.

> this thread references at least two different methods No, as the sidethread comment notes, there is only one way you can compute quartiles. You seem to be arguing that the correct thing to do is to impute them, and that calculating them is such a deviant practice that it would need to be specially remarked on.

Isn't this what i was saying from the beginning? Box plots are made for visualizing generalized normal distributions and nothing else. And now people in this thread argue you can calculate them from something else. Not sure if you are replying to the right post.

That might be what you were saying from the beginning, but the only thing that that would establish is that you're completely out of touch with reality. Box plots are made for visualizing quartiles.

Your theory would imply, among other things, that the median line going through the box part of a box plot always divides it in half, which obviously is not the case.

Re: I've stopped using box plots (2021)

#227

Post by sigmoid10 »

imachine1980_ wrote: post #180
sigmoid10 wrote: post #12

>box plots always make distributions look bell shaped I feel like this is where the confusion stems from for the author and everyone else here. Box plots don't make anything bell shaped (they don't change the distribution), they assume that your data follows a bell/gaussian shape. This is correct in cases where the central limit theorem can be applied (which is almost everywhere) - but when that is not the case, the…

Could you please elaborate on the reason? I assume it’s related to a unique null derivative instead of multiple maxima, but I couldn’t find any papers or information on this. Additionally, I find the article informative but believe it could be improved with this clarification. As someone who has worked with data analytics but is not a mathematician or actuary, I know people who probably review these types of graphs.…

>related to a unique null derivative instead of multiple maxima

I think the word you're trying to use is "bimodal" and yes, that is one example where the author's reasoning fails. But it's not the only one.

>I couldn’t find any papers or information on this.

You said you have no formal higher education in mathematics - how would you even go about finding (let alone understanding) papers? Regardless, just to be clear, this is not something you would learn from papers but from introductory textbooks and university courses. Everyone who has to deal with statistics in science needs to go through a whole lot of extra education exactly because there are many pitfalls like this.

>it is essential to check the underlying data distribution to avoid being misled by the information

That is another half-truth that everyone on the outside seems to agree on, but it is useless in practice. What do you do if the underlying data is not accessible. And what if you don't have the means to process it for every paper you read (which is what usually happens)? Then you have to rely on the actual tricks of the trade, which will come naturally if you worked with tons of statistics before. There are lots of telltale signs that let you spot bad analyses by only looking at a plot or summary chart. Granted, you won't catch all of them, but it often takes real malice and deep statistical competence on the author's side to cover up these things.

Re: I've stopped using box plots (2021)

#228

Post by blueflow »

thaumasiotes wrote: post #226
blueflow wrote: post #217

Earlier quoted context omitted.

Isn't this what i was saying from the beginning? Box plots are made for visualizing generalized normal distributions and nothing else. And now people in this thread argue you can calculate them from something else. Not sure if you are replying to the right post.

That might be what you were saying from the beginning, but the only thing that that would establish is that you're completely out of touch with reality. Box plots are made for visualizing quartiles. Your theory would imply, among other things, that the median line going through the box part of a box plot always divides it in half, which obviously is not the case.

No? Exponential Gaussian?

Whatever you do, you should explain first what you do that your whiskers stay meaningful and are not just whatever randomness your outliers produced.

Re: I've stopped using box plots (2021)

#229

Post by kazinator »

rcxdude wrote: post #40
sigmoid10 wrote: post #12

>box plots always make distributions look bell shaped I feel like this is where the confusion stems from for the author and everyone else here. Box plots don't make anything bell shaped (they don't change the distribution), they assume that your data follows a bell/gaussian shape. This is correct in cases where the central limit theorem can be applied (which is almost everywhere) - but when that is not the case, the…

Nah, there's nothing in a box plot which assumes a bell-shape. It does, however just visualise the parameters which reasonably well characterize a smooth single-mode distribution regardless of the underlying distribution. So it's a valid criticism of using box plots, especially when the alternatives can just as well visualise a bell-shaped distribution, as well as showing when it is not.

The problem is that the four quantile groups contain equal numbers of items, but are not represented by equal areas, even if we replace the whiskers with a bar of the same width as the box.

The bottom whisker contains 25% of the data, yet is just a thin line, which can furthermore be arbitrarily short.

It really is a dumb visual presentation.

The only way to use it is to recover the five parameters from it, and then stop looking at it.

For that purpose, a QR code would be just as good, if not better. You'd need a device with a camera to get the parameters (but "everyone" has that now), and when you're looking at it with your bare eyes, it doesn't tell you any visual lie.

Re: I've stopped using box plots (2021)

#230

Post by seanhunter »

hoseja wrote: post #223
seanhunter wrote: post #212

Earlier quoted context omitted.

The summary is she is saying you almost always want to show one of two things (and not both): 1) To show the distribution, in which case just the histogram arranged horizontally in the traditional fashion is far better than a violin plot with 2 copies of the histogram vertically and some extra quartile stuff tacked on, especially since lots of standard libraries to do violin plots do kde with very extreme smoothing s…

3) They look like THAT

Well yes.
Post reply on HN