Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parquet-WASM: Rust-based WebAssembly bindings to read and write Parquet data

github.com

Post reply on HN
Print view View original

1–10 of 136 posts

Re: Parquet-WASM: Rust-based WebAssembly bindings to read and write Parquet data

#3

Post by leeoniya »

in my [albeit outdated] experience ArrowJS is quite a bit slower than using native JS types. i feel like crossing the WASMJS boundary is very expensive, especially for anything other than numbers/typed arrays.

what are people's experiences with this?

Re: Parquet-WASM: Rust-based WebAssembly bindings to read and write Parquet data

#4

Post by leeoniya »

jasonjmcghee wrote: post #2

Seeing as the popular alternative here would be DuckDB-WASM, which (last time I checked) is on the order of 50MB, this is comparatively super lightweight.

i think duckdb-wasm is closer to 6MB over wire, but ~36MB once decompressed. (see net panel when loading https://shell.duckdb.org/)

the decompressed size should be okay since it's not the same as parsing and JITing 36MB of JS.

Re: Parquet-WASM: Rust-based WebAssembly bindings to read and write Parquet data

#5

Post by ingenieroariel »

leeoniya wrote: post #3

in my [albeit outdated] experience ArrowJS is quite a bit slower than using native JS types. i feel like crossing the WASM JS boundary is very expensive, especially for anything other than numbers/typed arrays. what are people's experiences with this?

I'll let Kyle chime in but I tested it a few months ago with millions of polygons on an M2 16GB of RAM laptop and it worked very well.

There is a library by the same author called lonboard that provides the JS bits inside JupyterLab. https://github.com/developmentseed/lonboard

I think it is based on the Kepler.gl / Deck.gl data loaders that go straight to GPU from network.

Post reply on HN