Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I've stopped using box plots (2021)

nightingaledvs.com

Post reply on HN
Print view View original

171–180 of 258 posts

Re: I've stopped using box plots (2021)

#171

Post by kaitai »

iainmerrick wrote: post #130

Lots of people defending box plots here -- a lot more than I expected! What I don't see is anyone saying "box plots are useful because they're the best kind of chart for [specific use case]". I can't off-hand think of any situation where I'd rather see a box plot than a strip plot or violin plot. When and why would you want to summarise the data so coarsely and visualize it so un-intuitively?

I deal with a lot of business people who have processes that rely on 15th/85th percentile, or 25th/75th percentile. They want to see the median, the low/high percentiles, the max/min or outliers, and they don't want to see all the data points jittered in between. It's just overwhelming extraneous information. They in fact like tables with those numbers written down, but they want to compare ten different (time series of historical prices for different markets) and see it on one Powerpoint slide. The box plot allows a fast visual comparison of medians and other key percentiles (label the plot with the percentiles if you're doing something non-standard!). With jitter or violin they get hung up on weird random stuff and it derails meetings.

Important caveats: the generating processes for all these quantities are the same in a physical sense, so they are comparable. All the distributions are roughly lognormal-ish, so they are single-peaked distributions, as folks are discussing here. The point of the visualization in theses cases is not to understand the properties of the distribution per se, it's to show the important percentiles because they have business implications.

Re: I've stopped using box plots (2021)

#172

Post by blueflow »

munch117 wrote: post #102
blueflow wrote: post #100

Earlier quoted context omitted.

How else would you calculate the quartiles to render the boxes?

Count data points in each quartile. You can do that for any sortable data, independent of distribution.

On second thought, this method makes the outer brackets / whiskers pretty much useless since their position is determined by the largest outliers, which is quite much random.

Re: I've stopped using box plots (2021)

#173

Post by crazygringo »

sigmoid10 wrote: post #12

>box plots always make distributions look bell shaped I feel like this is where the confusion stems from for the author and everyone else here. Box plots don't make anything bell shaped (they don't change the distribution), they assume that your data follows a bell/gaussian shape. This is correct in cases where the central limit theorem can be applied (which is almost everywhere) - but when that is not the case, the…

Yes.

A lot of people here are commenting that no, technically box plots don't assume any distribution. And I mean, technically you can ride from NYC to SF in a lawnmower.

But I completely agree that box plots shouldn't ever be used for anything but unimodal distributions similar enough to a bell/gaussian distribution.

All of the criticism of the article seems to be that they're misleading when the distribution is not bell/gaussian, e.g. bimodal.

To which my reply is, of course. Box plots shouldn't be used then. But if your distribution is bell/gaussian, they seem fine and I see no particular issue with them.

Re: I've stopped using box plots (2021)

#174

Post by sanderjd »

j_bum wrote: post #146
mkl wrote: post #21

The only advantage box plots had is that they can be drawn by hand. Now that computers are ubiquitous this is no longer valuable. Violin plots and bee swarm plots are better. Jittered strip plots can be okay if you're careful to avoid saturation (or more points added in the saturated region will disappear as they can't make it any darker).

I disagree about violin plots being better. Here is a great rant (borderline lecture) from Angela Collier on why they aren’t [0] [0] https://youtu.be/_0QMKFzW9fw?si=86mRAZRnFCBfSzw0

Could you summarize the criticisms in this (pretty long) video, and what she is proposing as a better alternative (beanplots? or is she criticizing those too?)? I couldn't figure it out from perusing the transcript.

I think it's useful to be able to compare the approximate shapes of histograms during exploratory data analysis. Is the thesis of this criticism that this isn't actually a useful thing to do, or that violin plots don't achieve this, or is it "just" an aesthetic argument?

Re: I've stopped using box plots (2021)

#175

Post by pocketsand »

blueflow wrote: post #128
munch117 wrote: post #123

Earlier quoted context omitted.

Perhaps I expressed myself poorly, and left room for misunderstanding, because I cannot possible imagine that we have any real disagreement on how to compute quartiles. Any set of numbers I give you, you can compute quartiles for it. There is no algorithm for doing that that breaks down if the numbers don't follow a normal distribution.

Look at this SVG from wikipedia: https://upload.wikimedia.org/wikipedia/commons/1/1a/Boxplot_... When you calculate the box plot using normal distribution parameters, the outliers are outside the outer bracket. If you split the dataset into 4 equal parts, the bracket will be larger because the outliers are still inside it. The methodologies are not equal. This thread is the first time i heard people do the "split dat…

As I'm sure you know, there are a lot of variations on how quantiles are calculated in various software. The 25th percentile, e.g., doesn't always line up with a value in the dataset, so sometimes nearest rank methods are used, otherwise a linearly interpolated data point, where interpolation is done in various ways.

In any event, none of these methods assume normality, or rely on CDFs of a normal curve.

If they did, every box plot would be symmetric.

The fact some people think that boxplots are constructed in such a way is a pretty good reason to take the author's article seriously as for how boxplots are confusing.

Re: I've stopped using box plots (2021)

#176

Post by sloowm »

SillyUsername wrote: post #151
sloowm wrote: post #129

Earlier quoted context omitted.

That background would make you explicitly unqualified to asses the quality of box plots as a visualization method. Box plots are used throughout various fields of research that are far less mathematical in nature.

Rubbish. They're used extensively in probability statistics and confidence intervals. Field of research has bugger all to do with it :tears:

You not understanding what my comment means is incredibly thematic.

Re: I've stopped using box plots (2021)

#177

Post by magicalist »

cjk2 wrote: post #111
munch117 wrote: post #108

Earlier quoted context omitted.

A plot that requires the reader to be perfectly qualified is a bad plot.

They teach this to 15 year olds in the UK. If it's a bad plot, perhaps some introspection is required...

They also teach pie charts and use color scales with non-uniform brightness. Just because it's possible to read a plot doesn't make it a good plot.

Re: I've stopped using box plots (2021)

#178

Post by cycomanic »

canjobear wrote: post #157
thaumasiotes wrote: post #138

Earlier quoted context omitted.

And the smooth mollification will look like...?

It looks like a spike, not a bell.

A spike is not smooth (typically meaning continuous in the variable and its first derivative), which was one of the conditions.

Re: I've stopped using box plots (2021)

#179

Post by WWWWH »

rcxdude wrote: post #41
treflop wrote: post #34

Earlier quoted context omitted.

I agree. The author simply used the wrong chart. The author's example has a bimodal distribution (TWO peaks) and chooses a type of chart that has ONE peak (a box plot). A little baffling tbh.

Well, to start with, how would you determine that about your distribution in the first place? And if that works well enough, why use a box plot afterwards?

Yes, exactly! Just plot all the bloody data and be done with it. No one is doing this by hand anymore so it is no extra work.

To my mind, if you have a genuine EDA attitude you plot it all.

Re: I've stopped using box plots (2021)

#180

Post by imachine1980_ »

sigmoid10 wrote: post #12

>box plots always make distributions look bell shaped I feel like this is where the confusion stems from for the author and everyone else here. Box plots don't make anything bell shaped (they don't change the distribution), they assume that your data follows a bell/gaussian shape. This is correct in cases where the central limit theorem can be applied (which is almost everywhere) - but when that is not the case, the…

Could you please elaborate on the reason? I assume it’s related to a unique null derivative instead of multiple maxima, but I couldn’t find any papers or information on this.

Additionally, I find the article informative but believe it could be improved with this clarification. As someone who has worked with data analytics but is not a mathematician or actuary, I know people who probably review these types of graphs. Now, I understand that it is essential to check the underlying data distribution to avoid being misled by the information, even if the source and axes seem trustworthy

Post reply on HN