Live data from Hacker News

Perspective: Streaming pivot visualization via WebAssembly

github.com

81–90 of 141 posts

Re: Perspective: Streaming pivot visualization via WebAssembly

#81
post #37

If anyone finds this project exciting and is interested in learning more about working on Open Source at J.P.Morgan, feel free to send me a message - we are always looking to hire experienced, passionate talent!

I would like to work on cakeshop-- how do I get on touch with you?

For the blockchain projects, please email quorum_info@jpmorgan.com

Re: Perspective: Streaming pivot visualization via WebAssembly

#83
post #55

Earlier quoted context omitted.

Does it support: virtualization (showing only a view of full dataset that is updated from server on scroll) Pivoting (multiple levels/hierarchies on rows and on columns) Apart from features what part did you consider was the biggest hurdle? What browsers did you target?

It supports virtualization, and it is utilized by both plugins (Hypergrid and Highcharts). N levels of pivoting on both axes. Should work on all browsers - if not, please open an issue!

Is there a way to drill down a node dynamically, by clicking on row header during pivoting, and then see its children?

Re: Perspective: Streaming pivot visualization via WebAssembly

#86
post #2

I wrote the C++ part of this codebase, would be happy to answer any questions about it here.

Did you experiment with doing it JS-only prior to the C++ version? If so, what kind of performance increase did you see with WebAssembly?

I'd also be interested in some performance comparisons with a library like CrossFilter [1]. Does the improvement outweigh the penalties of crossing the JS/WebASM boundary?

[1] http://square.github.io/crossfilter/

Re: Perspective: Streaming pivot visualization via WebAssembly

#87

Earlier quoted context omitted.

Did you experiment with doing it JS-only prior to the C++ version? If so, what kind of performance increase did you see with WebAssembly?

I'd also be interested in some performance comparisons with a library like CrossFilter [1]. Does the improvement outweigh the penalties of crossing the JS/WebASM boundary? [1] http://square.github.io/crossfilter/

The boundary-crossing is definitely the bottleneck right now. We are currently putting alot of work into the Apache Arrow support specifically to avoid this crossover, which will allow us to send data from the server in binary and avoid parsing in the browser.

Re: Perspective: Streaming pivot visualization via WebAssembly

#88
post #36

Earlier quoted context omitted.

Even outside of C/C++, Emscripten seems to be required in some fashion for just about every "compile X to webasm". Which i'm not that happy about as it's quite heavy and takes a long time to compile, which makes it harder for newcomers to try it out because it requires a fairly large time investment to even compile hello world.

> Even outside of C/C++, Emscripten seems to be required in some fashion for just about every "compile X to webasm". One exception to this is Rust's new "wasm32-unknown-unknown" target, which uses LLVM to directly generate wasm files without going through emscripten: https://www.hellorust.com/news/native-wasm-target.html The easiest way to use it for larger projects is probably via https://github.com/koute/cargo-web…

Yeah I'm super excited about Rust's new webasm compiler, as well as some new-ish stuff with .Net that works without Emscripten.

I get the ecosystem is super young, but it was honestly what kept me from playing with Webasm for a while, because each time I'd sit down to play with it I'd be in for a 3 hour compile after spending an hour getting my windows machine to have the right compilers and not stomping over other things I have setup on here.

Re: Perspective: Streaming pivot visualization via WebAssembly

#89
post #70

Presumably, the point of using WASM and service workers is for performance, but I see no benchmarks. I also have trouble imagining that this actually improves performance, unless you’re doing all of your compute in the browser and it’s CPU bound (this seems like bad design). What is the performance of doing it this way vs. without WASM and workers? Until we see numbers, this smells like a recruiting play.

For ticking data being pushed into bespoke pivots, running this on the backend and, say, pushing renders to the frontend isn't generally better. You're not likely to be sharing much of the grunt. Also, latency will be an issue so you need to be pretty careful with GC runtimes that aren't optimised for latency.

> What is the performance of doing it this way vs. without WASM and workers?

@dman says in another thread that the original was in Python and this was 10x faster. If this met their performance goals, it does the job. It may be possible that another way is better, for some definition of better, but who cares if it works as required?

> Until we see numbers, this smells like a recruiting play.

It's presumably OK not to use it if you don't want to.

And what if they think this may help recruiting? Seems like a reasonable trade to me.

Re: Perspective: Streaming pivot visualization via WebAssembly

#90
post #55

Earlier quoted context omitted.

It supports virtualization, and it is utilized by both plugins (Hypergrid and Highcharts). N levels of pivoting on both axes. Should work on all browsers - if not, please open an issue!

Is there a way to drill down a node dynamically, by clicking on row header during pivoting, and then see its children?

[deleted]
Post reply on HN