Python Data Visualization 2018: Why So Many Libraries?

I used it for charts in a paper recently, since it includes swarm plots. I hit a problem when overlaying certain types of plot on the same axes (I think it was swarm plots on top of box plots, so I could show every data point as well as the quartiles). The problem was that the data would end up shifted, so the x axis labels weren't correct, even though each plot on its own would work fine when the other was commented-out. The only reason I noticed this was because I spotted that a peak I knew occurred at x=13 was showing up at x=14.

Although it took a specific set of circumstances to trigger, I thought this was still a serious problem since it causes data to be misrepresented, and it doesn't cause any warning, etc. that something is wrong. I made a minimal example script, opened a github issue ( https://github.com/mwaskom/seaborn/issues/1409 ), where the library author insulted me and locked the issue. I sent them a followup email, to explain that I was not after "free tech support" (I'd already worked around the issue for my paper by 'faking' the labels), I wanted to help improve the library so that others would avoid having incorrect plots (especially those not lucky enough to spot it like I was) and how I'd already spent considerable time narrowing down the problem to the minimal example, as evidence that I wasn't trying to be a freeloader. I also began my email with an apology for using this side channel, and that I wouldn't contact them again unless they consented to it. The author replied with more insults.

I didn't contact them again, as promised, but now I'm actively opposed to anyone using this project, due to the author's complete disregard for corruption of scientific data.

I first noticed this effect with Python web-frameworks years ago. There was always some new framework that worked better or differently for some use case. (There are 73 web-frameworks listed in the python.org wiki.)

A few more examples from pypi.org searches:

“Reed-Solomon” 84 packages

“Elliptic-curve” 535 packages

“Nearest neighbor” 408 packages

“Simplex” (as in simplex method) 29

“Django” 10,000+

I just noticed a HN post about a RAFT implementation in Haskell. Pypi.org says there are 22 hits for RAFT in the python package index.

I see how you got confused, but I was making two separate assertions.

Assertion 1: If you merge the library projects together, and strip out the redundancies between them, then you'll have the same number of contributors, now distributed over fewer total lines of code. So each contributor can learn more of the codebase. (Bus factor goes up.)

Assertion 2: If you refactor the resulting libraries to reduce the total complexity (i.e. reduce the API surface from that of the union of all the merged-in libs), then you can begin to strip out technical debt from the project from the outside in. By eliminating now-dead code, you remove places where bugs can arise, and you lower the number of dependencies (which could otherwise have been sources of API-breaking changes when they update.) Thus, the number of people needed to maintain the project goes down. So the same total functionality can then be maintained by fewer contributors. (You don't actually remove contributors; they just become reserve capacity, with each contributor able to be less-overworked for the same result.)

Alas, it's not so simple. How much actual overlap is there to merge in Bokeh and Matplotlib, for instance? MPL renders images in Python and has no JS component. Bokeh does all of its rendering in JavaScript! The Python API is mostly just a thin wrapper around BokehJS. MPL has no server component at all. Neither of the does what Datashader does for large data sets. Neither of them has a high level statistical charting API, that's only in Seaborn or Chartify. Merging all these things together would cost a fortune in time and money, and at the end of the day not actually reduce the total codesize to any appreciable degree.