Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I've stopped using box plots (2021)

nightingaledvs.com

Post reply on HN
Print view View original

241–250 of 258 posts

Re: I've stopped using box plots (2021)

#241

Post by sanderjd »

hoseja wrote: post #223
seanhunter wrote: post #212

Earlier quoted context omitted.

The summary is she is saying you almost always want to show one of two things (and not both): 1) To show the distribution, in which case just the histogram arranged horizontally in the traditional fashion is far better than a violin plot with 2 copies of the histogram vertically and some extra quartile stuff tacked on, especially since lots of standard libraries to do violin plots do kde with very extreme smoothing s…

3) They look like THAT

This I don't get, they usually look a lot nicer than other visualizations I see. What's the issue here?

Re: I've stopped using box plots (2021)

#242

Post by sanderjd »

SebastianKra wrote: post #195
sanderjd wrote: post #174

Earlier quoted context omitted.

Could you summarize the criticisms in this (pretty long) video, and what she is proposing as a better alternative (beanplots? or is she criticizing those too?)? I couldn't figure it out from perusing the transcript. I think it's useful to be able to compare the approximate shapes of histograms during exploratory data analysis. Is the thesis of this criticism that this isn't actually a useful thing to do, or that viol…

Her argument that convinced me, is that the same result can always be better represented with multiple histograms - z-stacked, side-by-side, 3D or ridgeline-plots (ridgeline plots look awesome). Check out her examples at 21:11. Compared to these alternatives, violin plots are comically bad.

I watched that part of the video and I just truly don't think any of the options you listed here are as easy to parse as a normal violin chart. They look like the kind of thing I'd see in a superficial infographic, not a serious analysis.

Re: I've stopped using box plots (2021)

#243

Post by sanderjd »

Aachen wrote: post #196
sanderjd wrote: post #174

Earlier quoted context omitted.

Could you summarize the criticisms in this (pretty long) video, and what she is proposing as a better alternative (beanplots? or is she criticizing those too?)? I couldn't figure it out from perusing the transcript. I think it's useful to be able to compare the approximate shapes of histograms during exploratory data analysis. Is the thesis of this criticism that this isn't actually a useful thing to do, or that viol…

The two other replies are her main point(s), but the video also spends some time on another issue that she labels as minor but I found interesting to hear the perspective on. I'll try to do it justice: They look like vulvas. We're all adults, it's not a problem typically, but given that it's an aesthetic choice (noticing how half of the chart conveys the same info without this property), why? And it does come up, lik…

I'm honestly not sure what else to say about this besides: that's stupid. If someone is making that kind of joke and/or looking at the women in the room for validation ... how embarrassing for that childish person.

Re: I've stopped using box plots (2021)

#244

Post by ColFrancis »

blueflow wrote: post #222
ColFrancis wrote: post #199

Earlier quoted context omitted.

For what it's worth, you've convinced me that my beloved box plots need to be explained if I want to use them again. The SVG you've provided clearly shows that the box plot splits the data in 4. The interquartile range (IQR) is clearly marked and it even has a comparison for what the standard deviation (variance) measure would be. Secondly, if the data truly came from a normal distribution, there are no outliers. Out…

When i said "splitting", i meant it like my parent explained: Basically sorting your datasets and then splitting into quarters. What you want to explain to me (IMHO to the wrong person) is the correct approach of calculating a mean and standard deviation and drawing the box from that. Lets stay with that (and thats what i said earlier in the thread) After i wrote the post you replied to, i realized that the pure "spl…

The quartiles are defined by doing the sorting and splitting algorithm. So if you want quartiles (or any other quantile generally) you need to calculate it that way. The mean and standard deviation (sigma) are fundamentally different, which is why the image you linked shows them to contrast against the quantiles.

If you want to represent the standard deviation with your box plot, you can calculate it using standard formulas, many maths libraries have them built in. I don't know how to plot it using any graphing package though. ggplot, plotly and matlab all use the quantiles (the ones I have experience with). Perhaps where ever you learned to read them as mean and standard devation has a reference you could use?

> They are too random to be meaningful. It does not make sense to draw a box plot from that.

This can be a problem. In practice, the distributions I see don't go too crazy and are bounded (production rates can't be negative and can't be infinite). I prefer to use the 10th and 90th percentiles which are well defined and better behaved for most distributions. I do make sure it's very clearly marked on each plot though as it's not standard. Using the 1.5 x IQR cutoff is no better though as when you have enough samples you find that the whiskers just travel out to the cutoff.

Re: I've stopped using box plots (2021)

#245

Post by ColFrancis »

pocketsand wrote: post #175
blueflow wrote: post #128

Earlier quoted context omitted.

Look at this SVG from wikipedia: https://upload.wikimedia.org/wikipedia/commons/1/1a/Boxplot_... When you calculate the box plot using normal distribution parameters, the outliers are outside the outer bracket. If you split the dataset into 4 equal parts, the bracket will be larger because the outliers are still inside it. The methodologies are not equal. This thread is the first time i heard people do the "split dat…

As I'm sure you know, there are a lot of variations on how quantiles are calculated in various software. The 25th percentile, e.g., doesn't always line up with a value in the dataset, so sometimes nearest rank methods are used, otherwise a linearly interpolated data point, where interpolation is done in various ways. In any event, none of these methods assume normality, or rely on CDFs of a normal curve. If they did,…

As a first pass definition it does well to explain the concept. Even if you're interpolating you will need to rank the samples and find the two nearest neighbours to interpolate between.

It serves to distance it from the moment-based statistics like mean and variance at least.

Re: I've stopped using box plots (2021)

#246

Post by fnordpiglet »

jncfhnb wrote: post #231
sigmoid10 wrote: post #218

Earlier quoted context omitted.

I't appears you don't understand the central limit theorem fully. You gave the definition you find in textbooks, but you don't see how it applies to real world measurements and already explains your question. I can only recommend to visit a university level statistics course at this point. Maybe you will understand when you actually deal with some real data. Then you will indeed see its consequences pop up everywhere…

The GP here did not claim a poisson was the same thing as a Gaussian. They also don’t look similar. As far as I can tell you’re making the introductory student error of thinking the central limit theorem means any sufficiently large sample makes a distribution look normal.

Yes precisely. I’m also frankly shocked at the condescension. If you weren’t making that typical mistake then please explain the uses you meant. I would rarely use a box plot for a CLT distribution. Why? I would most often use it with a population sample I want to get the distribution of. Yes the mean of those would be a Gaussian under the CLT but it’s not useful useful as such.

Most natural distributions are not Gaussian upon sampling even if they’re bell shaped. They often have fatter tails, model some complex process, etc. The box plot is sometimes deceptive as is demonstrated in the original link. I don’t think that’s easy to argue against as they provide a totally reasonable and common sample distribution and show it failing to be descriptive of the most important features.

I fail to see how the CLT even remotely addresses the concerns or obviates them in any useful sense. The CLT and box plots aren’t very often applied together for these reasons.

Re: I've stopped using box plots (2021)

#247

Post by seanhunter »

sanderjd wrote: post #241
hoseja wrote: post #223

Earlier quoted context omitted.

3) They look like THAT

This I don't get, they usually look a lot nicer than other visualizations I see. What's the issue here?

Watch the video to understand her perspective on this. I don't want to spoil them for you if you like the look and once seen it's hard to unsee.

Re: I've stopped using box plots (2021)

#248

Post by seanhunter »

sanderjd wrote: post #240
seanhunter wrote: post #212

Earlier quoted context omitted.

The summary is she is saying you almost always want to show one of two things (and not both): 1) To show the distribution, in which case just the histogram arranged horizontally in the traditional fashion is far better than a violin plot with 2 copies of the histogram vertically and some extra quartile stuff tacked on, especially since lots of standard libraries to do violin plots do kde with very extreme smoothing s…

This is helpful! Is there a different name for the version of this that doesn't include the summary statistics on the same graph? I think seeing the distributions at different x-axis values (in my work, nearly always in a time series), but including the summary statistics is not as important and I agree that it's noisy.

Vertical kernel density estimation plot maybe? I'm not 100% sure what you mean. It would just be a vertical histogram if you're not doing kde.

Re: I've stopped using box plots (2021)

#249

Post by Aachen »

sanderjd wrote: post #243
Aachen wrote: post #196

Earlier quoted context omitted.

The two other replies are her main point(s), but the video also spends some time on another issue that she labels as minor but I found interesting to hear the perspective on. I'll try to do it justice: They look like vulvas. We're all adults, it's not a problem typically, but given that it's an aesthetic choice (noticing how half of the chart conveys the same info without this property), why? And it does come up, lik…

I'm honestly not sure what else to say about this besides: that's stupid. If someone is making that kind of joke and/or looking at the women in the room for validation ... how embarrassing for that childish person.

Perhaps, but (1) that's apparently what happens nevertheless and (2) to be clear, I'm just (further) answering the question about what's contained in this super long video

Re: I've stopped using box plots (2021)

#250

Post by sanderjd »

seanhunter wrote: post #248
sanderjd wrote: post #240

Earlier quoted context omitted.

This is helpful! Is there a different name for the version of this that doesn't include the summary statistics on the same graph? I think seeing the distributions at different x-axis values (in my work, nearly always in a time series), but including the summary statistics is not as important and I agree that it's noisy.

Vertical kernel density estimation plot maybe? I'm not 100% sure what you mean. It would just be a vertical histogram if you're not doing kde.

I just mean the "violin" part - yes, just a vertical histogram, but centered - without including the "hard to read" summary metrics on top of it.
Post reply on HN