Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fastplotlib: GPU-accelerated, fast, and interactive plotting library

medium.com

Post reply on HN
Print view View original

181–190 of 190 posts

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#181

Post by kushalkolar »

enriquto wrote: post #180
stackedinserter wrote: post #176

Earlier quoted context omitted.

You can't draw a proper plot with 3 million points at 30 fps, unless you cut corners, like not showing distribution of data (showing black rectangle when there's internal structure) or skipping peaks, like many plotting tools do, e.g. Grafana.

Of course you can! The screen of my laptop has nearly 3 million points (2160x1350) and I can do a fair amount of processing on each of its pixels, with one CPU thread, and still be above 30fps. A naive plotting method that loops over all the points and puts them into a grid will work without problem. Try it yourself!

Setting the value of a pixel in an image is very different from drawing objects like lines, this is a good introduction: https://graphicscompendium.com/intro/01-graphics-pipeline

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#182

Post by unnah »

almarklein wrote: post #145
wodenokoto wrote: post #142

How is it compared to HoloViz?[1] I followed one of their online workshops, and it feels really powerful, although it is a bit confusing which part of it does what (it's basically 6 or 7 projects put together under an umbrella) [1] https://holoviz.org/

One big difference is that Fastplotlib is based on GPU tech, so its capable of rendering much larger datasets interactively.

How much larger? Holoviz includes the datashader library for GPU-based rendering, and here is an example with 10 million points: https://examples.holoviz.org/gallery/nyc_taxi/nyc_taxi.html

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#183

Post by enriquto »

kushalkolar wrote: post #181
enriquto wrote: post #180

Earlier quoted context omitted.

Of course you can! The screen of my laptop has nearly 3 million points (2160x1350) and I can do a fair amount of processing on each of its pixels, with one CPU thread, and still be above 30fps. A naive plotting method that loops over all the points and puts them into a grid will work without problem. Try it yourself!

Setting the value of a pixel in an image is very different from drawing objects like lines, this is a good introduction: https://graphicscompendium.com/intro/01-graphics-pipeline

Ultimately, objects are always drawn in the screen by setting pixels into it. Plotting a point by setting a pixel is entirely reasonable, and can be indeed done directly, in realtime, for several millions of points. I just tested the C program below, compiled with gcc without optimizations, and it gives about 80 fps for three million points (on my 6 year-old thinkpad). My point: CPUs are ridiculously fast, and you can indeed do a lot of large-scale data visualization without need to meddle with the GPU.

    #define FPS 80

    void plot_points(
                    float *o,  // output raster array (w*h)
                    int w,     // width of raster
                    int h,     // height of raster
                    float *x,  // input point coordinates (2*n)
                    int n      // number of input points
                    )
    {
            // initialize the output raster
            for (int i = 0; i = 0 && p = 0 && q 
    int main(void)
    {
            int w = 1000;
            int h = 1000;
            int n = 3000000;
            float *x = malloc(2*n*sizeof*x);
            float *o = malloc(w*h*sizeof*o);
            for (int i = 0; i

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#184

Post by fransje26 »

jampekka wrote: post #173
fransje26 wrote: post #170

Earlier quoted context omitted.

Complaining about the inconsistencies of the matplotlib interface is pretty much beating a dead horse by now, and has been done repeatedly and in detail by others. The problems start as soon as you try doing something more than plt.plot(), and you get your first encounter with the maddening interface differences between a single figure plot and a multi-figure plot. And then it spirals out of control from there. There…

Sure there are some inconsistencies and legacy, but I wouldn't call that "horribly broken". You're probably referring to plots with subplots. Those indeed have issues, although mostly not because of the API. This has somewhat improved with the constrained layout, within the old API. There's also now GridSpec for more control. And for EDA those don't really matter much. There are some annoying differences when calling…

> You're probably referring to plots with subplots. Those indeed have issues, [..] There are some annoying differences when calling Axis methods vs the global functions [..]

When plotting is the basis of what a library does, and there are annoying differences encountered at such a very basic usage level, then it is not completely unreasonable to express some grievance about the syntax imposed to the user.. It is a frustrating user-experience to start encountering issues at such a fundamental level.

Tweaking plots, axis and layouts is tricky. Animating a plot with a bit of control is non-trivial, although I am prepared to concede that the two are different beasts.

My most recent annoyance was for something that seemed superficially easy: duplicating a left axis to a right axis, with a different label text, keeping the "original" grid and limits. Think of degree Kelvin on the left, and the equivalent in Celsius as a right axis. After more than 30 minutes of trying, I simply gave up as it was way beyond the amount of time I could justify spending on a single plot.

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#185

Post by kushalkolar »

enriquto wrote: post #183
kushalkolar wrote: post #181

Earlier quoted context omitted.

Setting the value of a pixel in an image is very different from drawing objects like lines, this is a good introduction: https://graphicscompendium.com/intro/01-graphics-pipeline

Ultimately, objects are always drawn in the screen by setting pixels into it. Plotting a point by setting a pixel is entirely reasonable, and can be indeed done directly, in realtime, for several millions of points. I just tested the C program below, compiled with gcc without optimizations, and it gives about 80 fps for three million points (on my 6 year-old thinkpad). My point: CPUs are ridiculously fast, and you ca…

OK now try to do this in 3D with arbitrary projections and interactivity! And guess what, you'd create a rendering engine :)

My earlier reply has a link to how GPUs actually push pixels to the screen.

There are also some excellent blog posts on how line rendering is done:

https://almarklein.org/triangletricks.html

https://almarklein.org/line_rendering.html

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#186

Post by roastedpeacock »

juliusbk wrote: post #77

This looks super cool! Looking forward to trying it. I think a killer feature of these gpu-plotting libraries would be if they could take torch/jax cuda arrays directly and not require a (slow) transfer over cpu.

Not sure about the memory transfer bottleneck and potential mitigations. But out of interest, how insurmountable would it be to 'retool' fastplotlib to use JAX acceleration instead of wgpu?

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#187

Post by almarklein »

unnah wrote: post #182
almarklein wrote: post #145

Earlier quoted context omitted.

One big difference is that Fastplotlib is based on GPU tech, so its capable of rendering much larger datasets interactively.

How much larger? Holoviz includes the datashader library for GPU-based rendering, and here is an example with 10 million points: https://examples.holoviz.org/gallery/nyc_taxi/nyc_taxi.html

I don't know Datashader that well, but from what I understand, it generates an image from a set of primitives (e.g. points), and then allows you to interactively inspect that image. It does not re-render the points on every frame like Fastplotlib/Pygfx does.

Depending on your GPU, you can render say 1-50 million points smoothly. Also see e.g. https://github.com/pygfx/pygfx/discussions/819

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#188

Post by kushalkolar »

crazygringo wrote: post #70
kushalkolar wrote: post #61

Earlier quoted context omitted.

Thanks Ivo! Just to add on, colab is weird and not performant, this PR outlines our attempts to get jupyter-rfb working on colab: https://github.com/vispy/jupyter_rfb/pull/77

Thanks. Yeah I've been baffled as to why just interactive Matplotlib with a Colab kernel is so slow. The Colab CPU is fast (enough), the network is fast, I haven't been able to figure out where the bottleneck is either.

I just remembered, I think there is something weird with Google's servers or the network because performance was very poor even with a custom Google Cloud instance running jupyterlab, see this: https://github.com/vispy/jupyter_rfb/issues/95#issuecomment-...

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#189

Post by kushalkolar »

paddy_m wrote: post #102
kushalkolar wrote: post #61

Earlier quoted context omitted.

Thanks Ivo! Just to add on, colab is weird and not performant, this PR outlines our attempts to get jupyter-rfb working on colab: https://github.com/vispy/jupyter_rfb/pull/77

Is google colab slower than an equivalently powerful kernel running on a remote jupyter kernel? Are you running into network problems, or is it something specific to colab?

I just commented above, see this: https://github.com/vispy/jupyter_rfb/issues/95#issuecomment-...

Re: Fastplotlib: GPU-accelerated, fast, and interactive plotting library

#190

Post by stackedinserter »

enriquto wrote: post #183
kushalkolar wrote: post #181

Earlier quoted context omitted.

Setting the value of a pixel in an image is very different from drawing objects like lines, this is a good introduction: https://graphicscompendium.com/intro/01-graphics-pipeline

Ultimately, objects are always drawn in the screen by setting pixels into it. Plotting a point by setting a pixel is entirely reasonable, and can be indeed done directly, in realtime, for several millions of points. I just tested the C program below, compiled with gcc without optimizations, and it gives about 80 fps for three million points (on my 6 year-old thinkpad). My point: CPUs are ridiculously fast, and you ca…

You're plotting individual points here, not a proper data graph. Even if you need a cloud of points, it's not enough, since you need to have different sizes and types of points that may have different sizes based on another data column, and definitely need to be drawn with antialiasing, even if they're simple squares.

Then, to draw something like this imgur.com/a/mXvEBzl (ADS-B data, ~10 million points iirc), you need to connect points with (antialiased) lines, where individual pixel should be blended into plot with respect of line opacity. Also, lines can be of different thickness, so it multiplies your `o[w x p+p] += 1` again.

I'm not even talking about multiple layers that are quite standard.

I use my own plotting app, it takes a lot more than just slap a bunch of points into "float *o". Try to write your own, you will figure it out pretty quickly, unless you're ok with black blobs that resemble the input data.

Post reply on HN