Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perspective: Streaming pivot visualization via WebAssembly

github.com

Post reply on HN
Print view View original

111–120 of 141 posts

Re: Perspective: Streaming pivot visualization via WebAssembly

#111

Post by jarpineh »

texodus wrote: post #45
jarpineh wrote: post #17

Earlier quoted context omitted.

Briefly in the animation I see there's hierachical (or multi-dimensional) data used. Could you tell me about what sort of data (statistics, monitoring) this supports? Also, what methods are available for getting data streamed [into Perspective]? I'm skimming through the docs, but am not quite getting it... [edited for clarity]

Right now, you can call a load() or update() method on an engine or widget to pass JSON, CSV or Apache Arrow data (as an ArrayBuffer) to the engine. Apologize for the documentation clarity - this is our first release and very much under active construction!

Ok, thank you for this. I'll have to see some time about that Arrow support. JSON and CSV is already plenty enough.

I didn't have time for a proper look yet, so no worries. If at all possible post the code and data (or something like it) that is in those animations you have. That would perhaps allow users to dive in by tinkering here and there.

Re: Perspective: Streaming pivot visualization via WebAssembly

#112

Post by rch »

testnet wrote: post #68
JumpCrisscross wrote: post #40

Earlier quoted context omitted.

> Finance people especially love this sort of thing - they do want to browse through entire datasets vs seeing higher level metrics in a lot of cases To expand on this, a lot of finance is searching out and understanding, normalizing or exploiting edge cases.

This is why excel is king. just give me a grid with all the data and a pivot table

Except when Excel won't manage nearly all the data. I'll sometimes use Igor Pro† before coding something up however.

† (only because I'm familiar with it - I'm positive there are better GUIs)

Re: Perspective: Streaming pivot visualization via WebAssembly

#113

Post by texodus »

hackcasual wrote: post #69
texodus wrote: post #9

Earlier quoted context omitted.

True - worth pointing out that this library gracefully falls back to ASM.js if your browser does not support WebAssembly, though.

Does it fallback if Window.WebAssembly doesn't exist, or does it fallback if WebAssembly.instantiateStreaming fails?

The former. In practice, it seems some runtimes are still a bit buggy - so we also revert to ASM.js if e.g. you are on an iPhone

Re: Perspective: Streaming pivot visualization via WebAssembly

#114

Post by texodus »

polskibus wrote: post #99
texodus wrote: post #87

Earlier quoted context omitted.

The boundary-crossing is definitely the bottleneck right now. We are currently putting alot of work into the Apache Arrow support specifically to avoid this crossover, which will allow us to send data from the server in binary and avoid parsing in the browser.

Can you elaborate on how that could work? Does arrow really allow for abstracting away the need for serialization even in JS - server scenarios? I though it was more of a shared memory data frame utility ?

There is an early arrow example in the examples package - superstore-arrow.html. The idea is that, instead of converting your data to internet->text->JSON->ArrayBuffer, you just keep the data in binary and write it directly into the C++ heap ArrayBuffer as-is. We currently do not read this in C++ directly for various reasons related to how emscripten allocates memory, but the general idea is the same.

Re: Perspective: Streaming pivot visualization via WebAssembly

#115

Post by h2g3yhogw8 »

ChicagoDave wrote: post #19

I love seeing new UX on Web Assembly. Someone needs to port or create a standard layout engine like WPF to build on this. If the tools were there, C# + VS Code + solid layout engine...I'd dump JS + HTML + CSS in a "flash".

my question is how do game engines do styling ?

Re: Perspective: Streaming pivot visualization via WebAssembly

#116

Post by LukeB42 »

dman wrote: post #96
LukeB42 wrote: post #95

Earlier quoted context omitted.

I see it's possible to convert to Numpy arrays. Is `perspective` also a Python library?

Yes for much of its life perspective has been a python library. It offers a streaming dataframe abstraction in Python. I believe the Python bindings have not been open sourced yet.

That sounds super useful. Does it create its own window as a Python library or am I missing something?

Re: Perspective: Streaming pivot visualization via WebAssembly

#117

Post by adamsea »

texodus wrote: post #76
adamsea wrote: post #57

First, let me say this looks awesome! I’m curious if your considered React.js for the presentation layer (i.e. the js/html5 parts) - on the surface it seems like it would be a good fit - and if so, what you saw as the benefits or drawbacks of it vs the approach you went with.

We chose to go with a Web Components based interface for compatibility across frameworks, but alot of where we go in the future will be determined by the expertise of the developers we hire to work on it, and the community if there is interest there.

I hadn’t realized you were using Web Components. That makes sense. Thank you.

Re: Perspective: Streaming pivot visualization via WebAssembly

#118

Post by dman »

LukeB42 wrote: post #116
dman wrote: post #96

Earlier quoted context omitted.

Yes for much of its life perspective has been a python library. It offers a streaming dataframe abstraction in Python. I believe the Python bindings have not been open sourced yet.

That sounds super useful. Does it create its own window as a Python library or am I missing something?

The Python bits are unreleased and the python usage is restricted to desktop visualizations, so no you are not missing anything.

Re: Perspective: Streaming pivot visualization via WebAssembly

#119

Post by infinite8s »

sciyoshi wrote: post #86
stupidcar wrote: post #34

Earlier quoted context omitted.

Did you experiment with doing it JS-only prior to the C++ version? If so, what kind of performance increase did you see with WebAssembly?

I'd also be interested in some performance comparisons with a library like CrossFilter [1]. Does the improvement outweigh the penalties of crossing the JS/WebASM boundary? [1] http://square.github.io/crossfilter/

In addition to what texodus has said below, Crossfilter only implements a small subset of what perspective provides. For example, no streaming (although there is a related project from the Heer lab that supports incremental updates - https://github.com/jheer/datavore), only a single level of grouping, only in 1 dimension, and you can only support 16 dimension fields at once (without increasing a constant in the codebase).

Re: Perspective: Streaming pivot visualization via WebAssembly

#120

Post by infinite8s »

polskibus wrote: post #65

Amazing, despite the fact that some heavy-lifting is done by highcharts and hypergriid. What's missing for this to effectively kill all typical BI tools like PowerBI, Tableau?

Expensive sales teams. Also neither of those handles streaming data.
Post reply on HN