Is there a best practice how to stream and plot large signal data (e.g. > 1M data points of multiple sine waves) from a Python backend (e.g. numpy + FastAPI) to frontend?
My current solution is: fetch ADC data, convert the bytes to base64 and embed it to JSON that will be send to the frontend. Frontend reverses this process and plot it to eCharts.
Show HN: ChartGPU – WebGPU-powered charting library (1M points at 60fps)
181–190 of 226 posts
Re: Show HN: ChartGPU – WebGPU-powered charting library (1M points at 60fps)
#182Earlier quoted context omitted.
Not true. Fill rate and memory speed is still a huge bottleneck. The issue is not “rubbish” but memory speed. It is almost always memory speed, cache, ram, disk etc. There is this misconception that if one uses js or c# to tell a gpu what to do it is somehow slower than rust. It only is if you crunching data but moving memory to the gpu and telling gpu to crunch is virtually identical.
PCIe 6.0 x16 delivers ~128 GB/s so the billion points can be loaded in milliseconds onto the GPU. The GPU's memory is much faster.
Most consumers dont have that and at 60 fps you are already maxing it out and more assuming os is doing nothing else. Bandwidth even on gpus is still the bottleneck.
Even then, when u write to a framebuffer directly in the gpu if the locations of the points are not contiguous you are thrashing. Rendering points very fast is still very much about reducing the data set down to bypass all the layers of memory walls.
Re: Show HN: ChartGPU – WebGPU-powered charting library (1M points at 60fps)
#183If you have tons of datapoints, one cool trick is to do intensity modulation of the graph instead of simple "binary" display. Basically for each pixel you'd count how many datapoints it covers and map that value to color/brightness of that pixel. That way you can visually make out much more detail about the data. In electronics world this is what "digital phosphor" etc does in oscilloscopes, which started out as just…
That digital phosphor effect is fascinating! As someone who works frequently with DSP and occasionally with analogue signals, it's incredible to see how you can pull out the carrier/modulation just by looking at (effectively) a moving average. It's also interesting to see just how much they have to do behind the scenes to emulate a fairly simple physical effect.
Re: Show HN: ChartGPU – WebGPU-powered charting library (1M points at 60fps)
#184"The core insight" ...
Re: Show HN: ChartGPU – WebGPU-powered charting library (1M points at 60fps)
#185Nice to see WebGPU experiments in data visualization field. But the description is certainly misleading.
All the optimizations mentioned except LTTB downsampling in compute shaders can be done in WebGL.
Web charts with > 1 M points and 60 FPS zooming/panning have been available since 2019. For example, here's a line chart with 100M points (100x more): https://lightningchart.com/lightningchart-js-demos/100M/
But still, love to see it. WebGPU will surely go forward slowly as these things naturally do, but practical experimentation is essential.
Re: Show HN: ChartGPU – WebGPU-powered charting library (1M points at 60fps)
#186Earlier quoted context omitted.
That works if more overdraw = more intensity is all you care about, and may very well be good enough for many kinds of charts. But with heat map plots one usually wants a proper mapping of some intensity domain to a color map and a legend with a color gradient that tells you which color represents which value. Which requires binning, counting per bin, and determining the min and max values.
Emm.. no, you just do one render pass to a temp framebuffer with 1 red channel, then another fragment shader maps it to an RGB palette.
Wait, does additional blending let you draw to temp framebuffers with high precision and without clamping? Even so you'd still need to know the maximum value of the temp framebuffer though.
Re: Show HN: ChartGPU – WebGPU-powered charting library (1M points at 60fps)
#187If this doesn't work for you on a reasonably recent version of Firefox, you can enable experimental (but in my experience quite stable) WebGPU support in `about:config` by setting `dom.webgpu.enabled` to true.
Re: Show HN: ChartGPU – WebGPU-powered charting library (1M points at 60fps)
#188What's the best way to get all those points from a backend into the frontend webgpu compute shader? There doesn't seem to be a communication mechanism that has minimal memcopy or no serialization/deserialization, the security boundary makes this difficult. I have a backend array of 10M i16 points, I want to get this into the frontend (with scale & offset data provided via side channel to the compute shader). As it st…
Not sure, but I solved a similar problem many years ago, and ended up concluding it was silly to send all the data to the client when the client didn't have the visual resolution to show it anyway. So I sampled it adaptively client-side by precomputing and storing multiple zoom-levels. That way the client-side chart app would get the points and you could zoom in, but you'd only ever retrieve about 1000-2000 points at…
Yeah I agree, I'd like to get an idea of the order-of-magnitude of difference between the two approaches by trying it out but realistically I don't think there's an easy way to get a i16 raw array into the browser runtime with minimal overhead (WebRTC maybe?)
Re: Show HN: ChartGPU – WebGPU-powered charting library (1M points at 60fps)
#189uPlot maintainer here. this looks interesting, i'll do a deeper dive soon :) some notes from a very brief look at the 1M demo: - sampling has a risk of eliminating important peaks, uPlot does not do it, so for apples-to-apples perf comparison you have to turn that off. see https://github.com/leeoniya/uPlot/pull/1025 for more details on the drawbacks of LTTB - when doing nothing / idle, there is significant cpu being…
Really appreciate you taking the time to look, Leon - uPlot has been a huge inspiration for proving that browser charts don't have to be slow. Both points are fair: 1. LTTB peak elimination - you're right, and that PR is a great reference. For the 1M demo specifically, sampling is on by default to show the "it doesn't choke" story. Users can set sampling: 'none' for apples-to-apples comparison. I should probably add…
Blind sampling like this makes it useless for real-world statistics of the kind your users care about.
And column-oriented data is a must. Look at Rlang's data frames, pandas, polars, numpy, sql, and even Fortran's matrix layout.
Also need specialized expicitly targetable support for Float32Array and Float64Array. Both API and ABI are necessary if you want to displace incumbents.
There is huge demand for a good web implementation. This is what it takes.
Am interested in collaborating.
Re: Show HN: ChartGPU – WebGPU-powered charting library (1M points at 60fps)
#190uPlot maintainer here. this looks interesting, i'll do a deeper dive soon :) some notes from a very brief look at the 1M demo: - sampling has a risk of eliminating important peaks, uPlot does not do it, so for apples-to-apples perf comparison you have to turn that off. see https://github.com/leeoniya/uPlot/pull/1025 for more details on the drawbacks of LTTB - when doing nothing / idle, there is significant cpu being…
Original Flot maintainer here. I once had to deal with many million data points for an application. I ended up mip-mapping them client-side. But regarding sampling, if it's a line chart, you can sample adaptively by checking whether the next point makes a meaningfully visible difference measured in pixels compared to its neighbours. When you tune it correctly, you can drop most points without the difference being not…
This is a good sampling transform to offer. Call it "co-domain awareness" or something.