Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fastplotlib: GPU-accelerated, fast, and interactive plotting library

medium.com

Post reply on HN
Print view View original

81–90 of 190 posts

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#81

Post by kushalkolar »

juliusbk wrote: post #77

This looks super cool! Looking forward to trying it. I think a killer feature of these gpu-plotting libraries would be if they could take torch/jax cuda arrays directly and not require a (slow) transfer over cpu.

Thanks! That is a great question and one that I've we've been battling with as well. As far as we know, this is not possible due to the way different contexts are set up on the GPU https://github.com/pygfx/pygfx/issues/510

tinygrad which I haven't used seems torch-like and has a WGPU backend: https://github.com/tinygrad/tinygrad

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#82

Post by [deleted] »

fpl-dev wrote: post #78
wtallis wrote: post #31

Earlier quoted context omitted.

> Eigendecomposition of the covariance matrix, essentially PCA, is probably the first non-trivial step in the analysis of any dataset For a sufficiently narrow definition of "dataset", perhaps. I don't think it's the obvious step one when you want to start understanding a time series dataset, for example. (Fourier transform would be a more likely step two, after step one of actually look at some of your data .)

Exactly that's a good example!

[deleted]

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#83

Post by juliusbk »

kushalkolar wrote: post #81
juliusbk wrote: post #77

This looks super cool! Looking forward to trying it. I think a killer feature of these gpu-plotting libraries would be if they could take torch/jax cuda arrays directly and not require a (slow) transfer over cpu.

Thanks! That is a great question and one that I've we've been battling with as well. As far as we know, this is not possible due to the way different contexts are set up on the GPU https://github.com/pygfx/pygfx/issues/510 tinygrad which I haven't used seems torch-like and has a WGPU backend: https://github.com/tinygrad/tinygrad

Yeah, I remember looking into it myself as well, and not finding any easy path. A shame.... Maybe there's a hard way to do it though :)

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#84

Post by rossant »

Spoilage4218 wrote: post #29
rossant wrote: post #21

Shameless plug: I'm actively working on a similar project, Datoviz [1], a C/C++ library with thin Python bindings (ctypes). It supports both 2D and 3D but is currently less mature and feature-complete than fastplotlib. It is also lower level (high-level capabilities will soon be provided by VisPy 2.0 which will be built on top of Datoviz, among other possible backends). My focus is primarily on raw performance, visua…

I have always admired your datoviz library from afar and check the vispy2/vispy2-sandbox libraries on GitHub every few months to check up on it. When do you think 'soon' is?? Really looking forward to it!

Thanks! The code is currently managed by Nicolas Rougier in a GitHub repository that will be made public next week. This repository hosts the "graphics server protocol" (GSP), an intermediate layer between Datoviz and the future high-level plotting API. For the latter, we’ll need community feedback to shape an API philosophy that aligns with VisPy users' needs—let's aim to publish a write-up this month.

Implementing the API on top of GSP should be relatively straightforward, as the core graphics-related mechanisms are handled by GSP/Datoviz. We've created a Slack channel for discussions—contact me privately if you'd like to join.

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#85

Post by hatthew »

sfpotter wrote: post #6

Very cool effort. That said, and it's probably because of the kind of work that I do, but I have almost never found the four challenges to be any kind of a problem for me. Although I do think there is some kind of contradiction there. Plotting (exploratory data analyis ("EDA"), really) is all about distilling key insights and finding features hidden in data. But you have to some kind of intuition about where the need…

For me, one of the most annoying things in my workflow is when I'm waiting for the software to catch up. If I'm making a plot, there's a lot of little tweaks I want to do to visually extract the maximum amount of information from a dataset. For example, if I'm making a histogram, I may want to adjust the number of bins, change to log scale, set min/max to remove outliers, and change the plot size on page. For the sake of the argument, let's say I'm working with a set of 8 slices of the dataset, so I need to regenerate 8 plots every time I make a tweak. My workflow is: Code the initial plots with default settings, run numpy to process the data, run matplotlib to display the data, look at the results, make tweaks to the code, circle back to step 2. In that cycle, "wait for matplotlib to finish generating the plots" can often be one of the longest parts of the cycle, and critically it's the vast majority of the cumulative time that I'm waiting rather than actively doing something. Drawing plots should be near instantaneous; there's an entire industry devoted to drawing complicated graphics in 16ms or less, I shouldn't need to wait >100ms for a single 2d grid with some dots and lines on it.

Matplotlib is okay, but there's definitely room for improvement, so why not go for that improvement?

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#86

Post by rossant »

fpl-dev wrote: post #54
klaussilveira wrote: post #18

Very cool to see imgui empowering so many different things.

We love imgui! Big thanks to the imgui devs, and Pascal Thomet who maintains the python bindings for imgui-bundle, and https://github.com/panxinmiao who made an Imgui Renderer for wgpu-py!

Imgui is awesome! Thanks for mentioning imgui-bundle—I hadn’t heard of it before, but it looks great! [1]

[1] https://github.com/pthom/imgui_bundle

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#87

Post by [deleted] »

sfpotter wrote: post #6

Very cool effort. That said, and it's probably because of the kind of work that I do, but I have almost never found the four challenges to be any kind of a problem for me. Although I do think there is some kind of contradiction there. Plotting (exploratory data analyis ("EDA"), really) is all about distilling key insights and finding features hidden in data. But you have to some kind of intuition about where the need…

[deleted]

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#88

Post by paddy_m »

jampekka wrote: post #20
paddy_m wrote: post #9

Earlier quoted context omitted.

I'd be curious to hear more about your EDA workflow. What I want for EDA is a tool that let's me quickly toggle between common views of the dataset. I run through the same analysis over and over again, I don't want to type the same commands repeatedly. I have my own heuristics for which views I want, and I want a platform that lets me write functions that express those heuristics. I want to build the inteligence into…

At least I usually do prefer to do the EDA plotting by writing and editing code. This is a lot more flexible. It's relatively rare to need other interactivity than zooming and panning. The differing approaches probably can be seen in some API choices, although the fastplotlib API is a lot more ergonomic than many others. Having to index the figure or prefixing plots with add_ are minor things, and probably preferable…

Writing and editting code is a lot more flexible, but it gets repetitive, and I have written the same stuff so many times. It's all adhoc, and it fixes the problem at the time, then it gets thrown away with the notebook only to be written again soon.

As an example, I frequently want to run analytics on a dataframe. More complex summary stats. So you write a couple of functions, and have two for loops, iterating over columns and functions. This works for a bit. It's easy to add functions to the list. Then a function throws an error, and you're trying to figure out where you are in two nested for loops.

Or, especially for pandas, you want to separate functions to depend on the same expensive pre-calc. You could pass the existing dict of computed measures so you can reuse that expensive calculation... Now you have to worry about the ordering of functions.

So you could put all of your measures into one big function, but that isn't reusable. So you write your big function over and over.

I built a small dag library that handles this, and lets you specify that your analysis requires keys and provides keys, then the DAG of functions is ordered for you.

How do other people approach these issues?

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#89

Post by sfpotter »

hatthew wrote: post #85
sfpotter wrote: post #6

Very cool effort. That said, and it's probably because of the kind of work that I do, but I have almost never found the four challenges to be any kind of a problem for me. Although I do think there is some kind of contradiction there. Plotting (exploratory data analyis ("EDA"), really) is all about distilling key insights and finding features hidden in data. But you have to some kind of intuition about where the need…

For me, one of the most annoying things in my workflow is when I'm waiting for the software to catch up. If I'm making a plot, there's a lot of little tweaks I want to do to visually extract the maximum amount of information from a dataset. For example, if I'm making a histogram, I may want to adjust the number of bins, change to log scale, set min/max to remove outliers, and change the plot size on page. For the sak…

I think this varies a lot depending on what you're doing.

I agree 100% that matplotlib is really slow and should be made to run as fast as humanly possible. I would add a (3) to my list above: optimize matplotlib!

OTOH, at least for what I'm doing, the code that runs to generate the data that gets plotted dominates the runtime 99% of the time.

For me, adjusting plots is usually the time waster. Hence point (2) above. I'd love to be able to make the tweaks using a WYSIWYG editor and have my plotting script dynamically updated. The bins, the log scale, the font, the dpi, etc, etc.

I think with your 8 slices examples above: my (2) and (3) would cover your bases. In your view, is the rest of matplotlib really so bad that it needs to be burnt to the ground for progress to be made?

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#90

Post by kkoncevicius »

paddy_m wrote: post #88
jampekka wrote: post #20

Earlier quoted context omitted.

At least I usually do prefer to do the EDA plotting by writing and editing code. This is a lot more flexible. It's relatively rare to need other interactivity than zooming and panning. The differing approaches probably can be seen in some API choices, although the fastplotlib API is a lot more ergonomic than many others. Having to index the figure or prefixing plots with add_ are minor things, and probably preferable…

Writing and editting code is a lot more flexible, but it gets repetitive, and I have written the same stuff so many times. It's all adhoc, and it fixes the problem at the time, then it gets thrown away with the notebook only to be written again soon. As an example, I frequently want to run analytics on a dataframe. More complex summary stats. So you write a couple of functions, and have two for loops, iterating over…

I work with R and not python, so some things might not apply, but this:

> [...] it fixes the problem at the time, then it gets thrown away with the notebook only to be written again soon.

Is one of the reasons I stopped using notebooks.

One solution to your problem might be to create a simple executable script that, when called on the file of your dataset in a shell, would produce the visualisation you need. If it's an interactive visualisation then I would create a library or otherwise a re-usable piece of code that can be sourced. It takes some time but ends up saving more time in the end.

If you have custom-made things you have to check on your data tables, then likely no library will solve your problem without you doing some additional the work on top.

And for these:

> Or, especially for pandas, you want to separate functions to depend on the same expensive pre-calc. [...] Now you have to worry about the ordering of functions.

I save expensive outputs to intermediate files, and manage dependencies with a very simple build-system called redo [1][2].

[1]: http://www.goredo.cypherpunks.su

[2]: http://karolis.koncevicius.lt/posts/using_redo_to_manage_r_d...

Post reply on HN