Live data from Hacker News

Perspective: Streaming pivot visualization via WebAssembly

github.com

91–100 of 141 posts

Re: Perspective: Streaming pivot visualization via WebAssembly

#91
post #38

Earlier quoted context omitted.

Did you experiment with doing it JS-only prior to the C++ version? If so, what kind of performance increase did you see with WebAssembly?

No, when I first prototyped it I had written a Python version initially. From memory the first C++ build was ~80x faster than the Python prototype.

80x faster than using Pandas? What if you just used the Numpy values array?

Re: Perspective: Streaming pivot visualization via WebAssembly

#92
post #52

Earlier quoted context omitted.

Not the op, but there's more to frontend development than JS. You need to account for the API of browsers, the DOM, CSS and the myraid of frameworks and libraries built for the specific UI requirements of frontend development (e.g. React). So even if your favorite language of choice becomes compilable to WebAssembly, you'll still be learning an entirely different way of development that will be completely unfamiliar…

A paradigm shift could happen and what's cool is that it could be entirely something new. This reminds me of the early days of the WWW when the first programming paradigm was Perl/CGI, then Cold Fusion, then Legacy ASP, and then ActiveX, and so on. The possibility of some company creating, from scratch, a new web development paradigm, given everything we know today, is exciting.

Java applets were supposed to provide that in the 90s. Netscape was going to actually embed Java in the browser as an alternative to JavaScript, but there wasn't enough time, so they just shipped with JS, and Java was provided via a plugin.

Re: Perspective: Streaming pivot visualization via WebAssembly

#93
post #55

Earlier quoted context omitted.

It supports virtualization, and it is utilized by both plugins (Hypergrid and Highcharts). N levels of pivoting on both axes. Should work on all browsers - if not, please open an issue!

Is there a way to drill down a node dynamically, by clicking on row header during pivoting, and then see its children?

Not through the UI yet, only in the engine API - but the pivoting itself is quite fast, so should still be suitable for drilling down "on the fly" so to stpeak

Re: Perspective: Streaming pivot visualization via WebAssembly

#94
post #38

Earlier quoted context omitted.

No, when I first prototyped it I had written a Python version initially. From memory the first C++ build was ~80x faster than the Python prototype.

80x faster than using Pandas? What if you just used the Numpy values array?

You will get me into trouble! I said 80x faster than the Python prototype. I did benchmark this vs Pandas over the years and depending on the usecase the two codebases trade blows with each other for static data. However for streaming datasets Perspective has a large advantage over Pandas since pandas does a full group by on every update, while perspective does its work incrementally using deltas.

Re: Perspective: Streaming pivot visualization via WebAssembly

#96
post #95
post #2

I wrote the C++ part of this codebase, would be happy to answer any questions about it here.

I see it's possible to convert to Numpy arrays. Is `perspective` also a Python library?

Yes for much of its life perspective has been a python library. It offers a streaming dataframe abstraction in Python. I believe the Python bindings have not been open sourced yet.

Re: Perspective: Streaming pivot visualization via WebAssembly

#98
post #87

Earlier quoted context omitted.

I'd also be interested in some performance comparisons with a library like CrossFilter [1]. Does the improvement outweigh the penalties of crossing the JS/WebASM boundary? [1] http://square.github.io/crossfilter/

The boundary-crossing is definitely the bottleneck right now. We are currently putting alot of work into the Apache Arrow support specifically to avoid this crossover, which will allow us to send data from the server in binary and avoid parsing in the browser.

bringing Apache Arrow in the browser alongside wasm is exciting to say the least! Amazing capabilities are coming to browsers...

Re: Perspective: Streaming pivot visualization via WebAssembly

#99
post #87

Earlier quoted context omitted.

I'd also be interested in some performance comparisons with a library like CrossFilter [1]. Does the improvement outweigh the penalties of crossing the JS/WebASM boundary? [1] http://square.github.io/crossfilter/

The boundary-crossing is definitely the bottleneck right now. We are currently putting alot of work into the Apache Arrow support specifically to avoid this crossover, which will allow us to send data from the server in binary and avoid parsing in the browser.

Can you elaborate on how that could work? Does arrow really allow for abstracting away the need for serialization even in JS - server scenarios? I though it was more of a shared memory data frame utility ?

Re: Perspective: Streaming pivot visualization via WebAssembly

#100
post #11
post #8

Earlier quoted context omitted.

I see it's possible to embed it in jupyterlab, it seems to do a lot: * grid format * graph format * actual pivoting of data Is that right? It's using webassembly so in effect to do the pivot you must have created functions that exist in pandas? Has anyone ported pandas to webassembly? Is the grid editable? What are you using to create your output, html or canvas or something? I realise I could look this stuff up but…

I believe you'd need to port Python to WebAssembly to make much use of pandas - unaware of any projects attempting the former. The grid is just a plugin for the excellent [Hypergrid]( https://github.com/openfin/fin-hypergrid ), which is editable, but you'd need to manually push those edits back to the engine for now.

As historical background: Python was one of the early emscripten asm.js experiments: https://github.com/kripken/emscripten/tree/master/tests/pyth...
Post reply on HN