Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

github.com

Post reply on HN
Print view View original

41–50 of 84 posts

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#41

Post by prasoonds »

maximilianroos wrote: post #32
prasoonds wrote: post #9

Earlier quoted context omitted.

Yes! I'm heartened to see people realize the the benefits of PRQL and chained transforms over vanilla SQL (with some caveats!). We'll definitely add a PRQL/SQL block to the UI soon (you will be able to toggle to select which one you'd like). On that point - PRQL doesn't natively support PIVOT statements. As a hack, we made our own flavor of PRQL with a PIVOT statement and we parse that to SQL CTEs to make it work for…

[PRQL dev here] Love what you've done and thanks for building on PRQL! We're very happy to add something like a PIVOT statement. In the long term, we'd have to think about how to make it work deeply in PRQL, since the column names are suddenly runtime dependent. In the short term, we should definitely make PRQL work for your case — it's important that there's an escape hatch for things that aren't natively supported…

Hey Max, love what you all doing with PRQL! It's a wonderful tool.

That makes sense. I ran into some old issues about Pivot but it seems it's not been implemented yet. And yes, s-strings wouldn't work - the DuckDB pivot statement looks like this:

  PIVOT ⟨dataset⟩ 
  ON ⟨columns⟩
  USING ⟨values⟩ 
  GROUP BY ⟨rows⟩
So, we'd have to pass the `dataset` so far into the pivot statement. This is where CTEs come in handy but happy to hear if there's a better solution :)

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#42

Post by prasoonds »

yawnxyz wrote: post #37
logical_person wrote: post #35

200mb of data is not a large file, and chromium tabs have a memory limit of something ridiculously low so actual large 20-100gb datasets render this useless.

Sometimes you need a scooter, sometimes you need a truck. I think a snappy interface for <1gb datasets is really neat and super useful for certain kinds of data

This echoes my thoughts exactly. Right now, we're actually more limited by the JS UI so a couple 100 MBs is the most you can do in a browser otherwise the UI becomes really slow. There's a lot of room for improvement - we're using React and that's causing a bunch of un-needed re-renders right now that we don't need. We probably need to create our own DAG based task management system and use Canvas to render everything - with all that, workflows on much larger files will hopefully become usable.

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#43

Post by prasoonds »

tobilg wrote: post #26

The DuckDB WASM space is really heating up! I released https://sql-workbench.com a few weeks ago, which can be used to query and visualize Parquet, CSV, JSON and Arrow data. There’s also a accompanying tutorial blog post at https://tobilg.com/using-duckdb-wasm-for-in-browser-data-eng...

Oh hey! I remember seeing your HN post several weeks ago, I think - it's a great tool, thanks for building this! It was definitely an inspiration for us :)

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#45

Post by prasoonds »

ThreeMoonsAreUp wrote: post #40

I enjoyed your demo video and played around with a CSV of my own. This is a cool project, but I have a (perhaps stupid) question: Why should I use this tool over Excel or Google Sheets? I don't mean this question to be a challenge. Just trying to understand which sorts of tasks would I be better off using Pretzel as opposed to a traditional spreadsheet tool.

Hey, absolutely - that's a perfectly fair question. I can give you my personal reasons:

- I don't have excel on a Mac so I need to use Google Sheets. Google Sheets crashes on any file over 30-40 MB (at least on my macbook air)

- I think this type of chained data transform is a much better (more code-like?) way of doing transforms. You can see exactly how you got to your end-result. Vs, for example, in Google Sheets, you're modifying data in-place and your transform history is only accessible via Undo and Redo. This is far more reproducible short of using a python script.

(I've worked as a data scientist for a while and I suppose that influences how I think about chained data transforms vs the in-place Excel way too!)

But this is just for now - the long term vision is: Being able to easily switch between visual no-code blocks, SQL and Python in the same browser-local "notebook"/"workbook" that's easily shareable. I think it doesn't take much convincing that such a tool would be more powerful than Excel for certain workflows!

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#46

Post by hitradostava »

Hey congrats on the Show HN. Local, browser based data exploration works for a lot of uses cases and is so much faster thancloud based tools. We've implemented something similar at https://addmaple.com/ - but with a graphical interface designed for rapid exploratory data analysis of large datasets.

Memory per tab can be an issue for really big files (1gb+) but we're exploring a transform to CBOR which allows us to free up JS memory, i.e. when parsing CBOR we can leave row level data as Uint8Array and it doesn't increase the JS memory overhead.

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#47

Post by prasoonds »

auntienomen wrote: post #44

This is quite nice. Any plans to add simpler ML tools? A sklearn plugin could be as valueable as LLM access.

Thank you! Yes, one of the items in the Roadmap is support for Pyodide (https://github.com/pyodide/pyodide) for running in-browser python on the results of each of the code blocks! This should allow most ML libs to be usable in-browser! This is pretty high-up on our priority list.

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#49

Post by tobilg »

prasoonds wrote: post #43
tobilg wrote: post #26

The DuckDB WASM space is really heating up! I released https://sql-workbench.com a few weeks ago, which can be used to query and visualize Parquet, CSV, JSON and Arrow data. There’s also a accompanying tutorial blog post at https://tobilg.com/using-duckdb-wasm-for-in-browser-data-eng...

Oh hey! I remember seeing your HN post several weeks ago, I think - it's a great tool, thanks for building this! It was definitely an inspiration for us :)

Thanks and congrats on the launch!

Re: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL

#50

Post by [deleted] »

prasoonds wrote: post #25
maxloh wrote: post #21

Earlier quoted context omitted.

It would be good to have EdgeQL too.

Oh this looks really interesting! I hadn't heard of EdgeDB/EdgeQL - I'll have to do a deeper read but at first brush, it looks like EdgeQL can be compiled down to SQL based on this HN comment: https://news.ycombinator.com/item?id=30296669 I'll take a look and if it's simply a matter of changing from the PRQL compiler to EdgeQL compiler, then, we should be able to have land in main in short order. It's not the highest…

[deleted]
Post reply on HN