Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

github.com

Post reply on HN
Print view View original

51–60 of 84 posts

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#51

Post by prasoonds »

dndn1 wrote: post #24

Looks great and thanks for sharing! You mentioned that you went to some length to implement pivots. How far do you think you will take the pivot feature and UI? For financial type usecases it's pretty much a requirement to be able to map many fields on X/Y, be able to collapse them in the the browser, and also show/control aggregations. Asking because many apps stop at the simplest level of pivot features and go to g…

Right - I used to work in GS and they had this really great internal table display tool where you could simply drag a column from the top to the left side and it would Pivot + Collapse all fields on it (plus allow multi level pivots). Then, you could look at the Pivot table OR do a drill-down to see the root-cause why a value was so high.

I really liked that interface and haven't really found anything near as useable yet. Perspective JS (a free library by JP Morgan, you can try it here: https://perspective.finos.org/block/) has some really cool functionality in this direction but it has its one data processing engine, query language, rendering engine etc so we couldn't have used that for this project.

But, this is an interesting thought - I definitely would want to see this functionality in Pretzel - the only question is how to prioritize this. If you know of any performant table libraries that support collapsible pivots out-of-the-box, I'd love to integrate that. Alternatively, we'll write our own!

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#52

Post by prasoonds »

prasoonds wrote: post #18
MarcoConverta wrote: post #14

Just tried it out - thanks for sharing. We're a sales tech startup and I've had to look through large CSVs with prospecting information in the past - anything over 10 MB crashes my browser. I use a Mac so I don't have Excel. This looks great for simple data manipulations. I tried out a large CSV I had, and it loaded without a problem. Quick bug report: The filter interface seems to be slow for me though with the larg…

Hey Marco, I'll take a look - filters theoretically should be fast, when you create a new filter, it simply reads does a `select * from table limit 1` to get column names I wasn't sure whether you could query DBs directly from the browser but looks like you can! ( https://github.com/alexanderguy/pgress ) - will add it to roadmap!

Replying as I couldn't edit:

Alright, did a bit of digging and the flamegraph points to a problem with the Table component. I'd hoped that BlueprintJS tables would be performant enough for our usecase but apparently not!

We'll try to move to canvas based rendering ASAP - that should fix any lags in the filter UI

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#54

Post by prasoonds »

hitradostava wrote: post #46

Hey congrats on the Show HN. Local, browser based data exploration works for a lot of uses cases and is so much faster thancloud based tools. We've implemented something similar at https://addmaple.com/ - but with a graphical interface designed for rapid exploratory data analysis of large datasets. Memory per tab can be an issue for really big files (1gb+) but we're exploring a transform to CBOR which allows us to fr…

Thanks! Maple looks really cool - really interesting demo video, too!

This is quite interesting - we've not explored really large files so far and being honest, we haven't thought that far either. Didn't know about CBOR! I will have to look deeper into how this can save on memory. I was wondering though, since WASM memory is limited to 4GB, if I have sufficiently large memory on my compute device, at least one tab should be able to handle 1gb+ files too, correct?

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#55

Post by chrisjc »

bbkane wrote: post #53

Wow! Is there any way this could support SQLite databases?

I haven't taken a close enough look at it yet, but duckdb supports Sqlite databases, so in theory yes.

However, using Sqlite requires a duckdb extension and i'm not sure if works for wasm, and if so enabled this project in particular.

https://duckdb.org/docs/guides/import/query_sqlite

But I believe that most of these duckdb web-IDEs will revolve around a couple of generalized used cases:

1. OLAP/analytical type workloads/queries. Sqlite is really more for OLTP/transactional workloads.

2. Querying datasets that are available on, targeted for blob/object storage like S3. Parquet, CSV, line-delimited JSON, etc

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#56

Post by prasoonds »

bbkane wrote: post #53

Wow! Is there any way this could support SQLite databases?

DuckDB supports SQLite through extensions (https://duckdb.org/docs/extensions/sqlite.html)

So, it should be pretty straightforward to let folks drop a SQLite file instead of a CSV file! I haven't been able to get extensions to work on WASM so far though but I will definitely take a look to see if we can make this work!

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#57

Post by hitradostava »

prasoonds wrote: post #54
hitradostava wrote: post #46

Hey congrats on the Show HN. Local, browser based data exploration works for a lot of uses cases and is so much faster thancloud based tools. We've implemented something similar at https://addmaple.com/ - but with a graphical interface designed for rapid exploratory data analysis of large datasets. Memory per tab can be an issue for really big files (1gb+) but we're exploring a transform to CBOR which allows us to fr…

Thanks! Maple looks really cool - really interesting demo video, too! This is quite interesting - we've not explored really large files so far and being honest, we haven't thought that far either. Didn't know about CBOR! I will have to look deeper into how this can save on memory. I was wondering though, since WASM memory is limited to 4GB, if I have sufficiently large memory on my compute device, at least one tab sh…

Thanks :-)

I've not done much profiling on DuckDB and what the overhead is - i.e. after the data is parsed how much memory is used. Would be really interesting to push it to the limit - or to explore not loading the entire file in, but only reading the relevant parts, but again that probably requires a conversion first, e.g. to parquet or some other column based storage format.

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#58

Post by prasoonds »

hitradostava wrote: post #57
prasoonds wrote: post #54

Earlier quoted context omitted.

Thanks! Maple looks really cool - really interesting demo video, too! This is quite interesting - we've not explored really large files so far and being honest, we haven't thought that far either. Didn't know about CBOR! I will have to look deeper into how this can save on memory. I was wondering though, since WASM memory is limited to 4GB, if I have sufficiently large memory on my compute device, at least one tab sh…

Thanks :-) I've not done much profiling on DuckDB and what the overhead is - i.e. after the data is parsed how much memory is used. Would be really interesting to push it to the limit - or to explore not loading the entire file in, but only reading the relevant parts, but again that probably requires a conversion first, e.g. to parquet or some other column based storage format.

that sounds like a fun project :) maybe I'll take a stab at it!

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#59

Post by dndn1 »

prasoonds wrote: post #51
dndn1 wrote: post #24

Looks great and thanks for sharing! You mentioned that you went to some length to implement pivots. How far do you think you will take the pivot feature and UI? For financial type usecases it's pretty much a requirement to be able to map many fields on X/Y, be able to collapse them in the the browser, and also show/control aggregations. Asking because many apps stop at the simplest level of pivot features and go to g…

Right - I used to work in GS and they had this really great internal table display tool where you could simply drag a column from the top to the left side and it would Pivot + Collapse all fields on it (plus allow multi level pivots). Then, you could look at the Pivot table OR do a drill-down to see the root-cause why a value was so high. I really liked that interface and haven't really found anything near as useable…

One that I'm watching is the Table mark in Graphic Walker [1].

There is a license thing about logos to note if you do use it in Pretzel [2].

On GW [1], Create Dataset -> Public Datasets -> Student Performance then change the Mark Type to Table you can play around. It hits the things that I mentioned pretty good! You might have similar issues as you have with Perspective though.

I make calculang [3] and I'm getting ready to plug into a lot of things. A good pivot table generalizes well for my needs (mainly quick feedback during DX).

Perspective is on the list but so is GW and Pretzel.

Perspective might suit my needs perfectly.

But I like DuckDB-WASM approach anyway so I hope you continue and I hope you nail it one way or another! :)

[1] https://graphic-walker.kanaries.net

[2] https://github.com/Kanaries/graphic-walker/issues/330

[3] https://calculang.dev

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#60

Post by antonycourtney »

prasoonds wrote: post #51
dndn1 wrote: post #24

Looks great and thanks for sharing! You mentioned that you went to some length to implement pivots. How far do you think you will take the pivot feature and UI? For financial type usecases it's pretty much a requirement to be able to map many fields on X/Y, be able to collapse them in the the browser, and also show/control aggregations. Asking because many apps stop at the simplest level of pivot features and go to g…

Right - I used to work in GS and they had this really great internal table display tool where you could simply drag a column from the top to the left side and it would Pivot + Collapse all fields on it (plus allow multi level pivots). Then, you could look at the Pivot table OR do a drill-down to see the root-cause why a value was so high. I really liked that interface and haven't really found anything near as useable…

Very impressive project and vision! Love the demo!

I am also ex-GS and worked on what I am fairly sure is the table display tool you're describing. I tried to carry the essential aspects of that work (multi-level pivots, with drill-down to the leaf level, and all interactive events and analytics supported by db queries) to Tad (https://www.tadviewer.com/, https://github.com/antonycourtney/tad), another open source project powered by DuckDb.

An embeddable version of Tad, powered by DuckDb WASM, is used as the results viewer in the MotherDuck Web UI (https://app.motherduck.com/).

If you're interested in embedding Tad in Pretzel, or leveraging pieces of it in your work, or collaborating on other aspects of DuckDb WASM powered UIs, please get in touch!

Post reply on HN