Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fastplotlib: GPU-accelerated, fast, and interactive plotting library

medium.com

Post reply on HN
Print view View original

31–40 of 190 posts

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#31

Post by wtallis »

macleginn wrote: post #30
wtallis wrote: post #24

Earlier quoted context omitted.

Aren't you missing the entire point of exploratory data analysis? Eigenfaces are an example of what you can come up with as the end product of your data exploration, after you've tried many ways of looking at the data and determined that eigenfaces are useful. Your whole third paragraph seems to be criticizing the core purpose of exploratory data analysis as though one should always be able to skip directly to the ne…

Eigendecomposition of the covariance matrix, essentially PCA, is probably the first non-trivial step in the analysis of any dataset. The idea in the comment above seems to be that it's more useful to combine some basic knowledge of statistics with simpler visualisation techniques, rather than to quickly generate thousands of shallower plots. Being able to generate thousands of plot is useful, of course, but I would a…

> Eigendecomposition of the covariance matrix, essentially PCA, is probably the first non-trivial step in the analysis of any dataset

For a sufficiently narrow definition of "dataset", perhaps. I don't think it's the obvious step one when you want to start understanding a time series dataset, for example. (Fourier transform would be a more likely step two, after step one of actually look at some of your data.)

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#32

Post by bee_rider »

mhh__ wrote: post #27
sfpotter wrote: post #6

Very cool effort. That said, and it's probably because of the kind of work that I do, but I have almost never found the four challenges to be any kind of a problem for me. Although I do think there is some kind of contradiction there. Plotting (exploratory data analyis ("EDA"), really) is all about distilling key insights and finding features hidden in data. But you have to some kind of intuition about where the need…

My hot take is that 3D plotting feels bad because 3D plots are bad. You can usually find some alternative way of representing the data

3D plots might be neat if there was some widespread way of displaying them. Unfortunately we can only make 2D projections of 3D plots on our computer screens and pieces of paper.

Maybe VR will change that at some point. :shrug:

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#33

Post by kkoncevicius »

wtallis wrote: post #24
kkoncevicius wrote: post #13

Earlier quoted context omitted.

I have to agree with your point about EDA. The library is neat, but even the example of covariance matrix animation is a bit contrived. Every pixel has a covariance with every other pixel, so sliding though the rows of the covariance matrix generates as many faces on the right as there are pixels in a photograph of a face. However the pixels that strongly co-vary will produce very similar right side "face" pictures.…

Aren't you missing the entire point of exploratory data analysis? Eigenfaces are an example of what you can come up with as the end product of your data exploration, after you've tried many ways of looking at the data and determined that eigenfaces are useful. Your whole third paragraph seems to be criticizing the core purpose of exploratory data analysis as though one should always be able to skip directly to the ne…

As a whole, of course you have a point - big visualisations when done properly should help with data exploration. However, from my experience they rarely (but not never) do. I think it's specific to the type of data you work with and the visualisation you employ. Let me give an example.

Imagine we have some big data - like an OMIC dataset about chromatin modification differences between smokers and non-smokers. Genomes are large so one way to visualise might be to do a manhattan plot (mentioned here in another comment). Let's (hypothetically) say the pattern in the data is that chromatin in the vicinity of genes related to membrane functioning have more open chromatin marks in smokers compared to non smokers. A manhattan plot will not tell us that. And in order to be able to detect that in our visualisation we had to already know what we were looking for in the first place.

My point in this example is the following: in order to detect that we would have to know what to visualise first (i.e. visualise the genes related to membrane function separately from the rest). But then when we are looking for these kinds of associations - the visualisation becomes unnecessary. We can capture the comparison of interest with a single number (i.e. average difference between smokers vs non-smokers within this group of genes). And then we can test all kinds of associations by running a script with a for-loop in order to check all possible groups of genes we care about and return a number for each. It's much faster than visualisation. And then after this type of EDA is done, the picture would be produced as a result, displaying the effect and highlighting the insights.

I understand your point about visualisation being an indistinguishable part of EDA. But the example I provided above is much closer to my lived experience.

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#34

Post by clewis7 »

theLiminator wrote: post #11

Do you have any numbers for the rough number of datapoints that can be handled? I'm curious if this enables plotting many millions of datapoints in a scatterplot for example.

Yes! The number of data points can range in the millions. Quite honestly, the quality of your GPU would be the limiting factor here. I will say, however, that for most use cases, an integrated GPU is sufficient. For reference, we have plotted upwards of 3 million points on a mid-range integrated GPU from 2017.

I will work on adding somewhere in our docs some metrics for this kind of thing (I think it could be helpful for many).

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#36

Post by clewis7 »

abdullahkhalids wrote: post #14

Is it possible to put the interactive plots on your website? Or is this a Jupyter notebook only tool.

See here: https://www.fastplotlib.org/ver/dev/user_guide/faq.html#what...

We are hoping for pyodide integration soon, which would allow fastplotlib to be run strictly in the browser!

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#37

Post by fpl-dev »

kkoncevicius wrote: post #13
sfpotter wrote: post #6

Very cool effort. That said, and it's probably because of the kind of work that I do, but I have almost never found the four challenges to be any kind of a problem for me. Although I do think there is some kind of contradiction there. Plotting (exploratory data analyis ("EDA"), really) is all about distilling key insights and finding features hidden in data. But you have to some kind of intuition about where the need…

I have to agree with your point about EDA. The library is neat, but even the example of covariance matrix animation is a bit contrived. Every pixel has a covariance with every other pixel, so sliding though the rows of the covariance matrix generates as many faces on the right as there are pixels in a photograph of a face. However the pixels that strongly co-vary will produce very similar right side "face" pictures.…

Hi, one of the other devs here. As the poster below pointed out what you're missing is that in this case we know that an eigendecomposition or PCA will be useful. However if you're working on matrix decomposition algorithms like us, or if you're trying to design new forms of summary matrices because a covariance matrix isn't informative for your type of data then these types of visualizations are useful. We broadly work on designing new forms of matrix decomposition algorithms so it's very useful to look at the matrices and then try to determine what types of decompositions we want to do.

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#40

Post by crazygringo »

Sounds really compelling.

But it doesn't seem to answer how it works in Jupyter notebooks, or if it does at all. Is the GPU acceleration done "client-side" (JavaScript?) or "server-side" (in the kernel?) or is there an option for both?

Because I've used supposedly fast visualization libraries in Google Colab before, but instead of updating at 30 fps, it takes 2 seconds to update after a click, because after the new image is rendered it has to be transmitted via the Jupyter connector and network and that can turn out to be really slow.

Post reply on HN