Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB-Wasm: Efficient analytical SQL in the browser

duckdb.org

Post reply on HN
Print view View original

11–20 of 62 posts

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#11

Post by nicoburns »

crimsoneer wrote: post #6

I'm still not sure I "get" the use case for DuckDB. From what I understand, it's like a nifty, in-memory SQL, but why is that better than just running PostGRES or Microsoft SQL server locally, where your data structures and tables and stuff have a lot more permanence? Like, my workflow is either I query an exiting remote corporate DB and do my initial data munging there, or get givne a data dump that I either work on…

DuckDB is columnar, so in theory a lot faster than Postgres or SQL server for Analytical workloads.

DuckDB is to Clickhouse, TimescaleDB, Redshift, etc as SQLite is to Postgres, MySQL, SQL Server.

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#12

Post by ankoh »

kf6nux wrote: post #8
crimsoneer wrote: post #6

I'm still not sure I "get" the use case for DuckDB. From what I understand, it's like a nifty, in-memory SQL, but why is that better than just running PostGRES or Microsoft SQL server locally, where your data structures and tables and stuff have a lot more permanence? Like, my workflow is either I query an exiting remote corporate DB and do my initial data munging there, or get givne a data dump that I either work on…

I don't fully get the use case either, but it's in a different category than Postgres or Microsoft SQL because it runs in the browser and can be made part of your web app.

DuckDB-wasm is targeting the browser so it's not directly competing with Pandas (that's the job of native DuckDB).

It's targeting use cases where you want to push analytical computation away from servers into the client (browser).

Lets me sketch 2 examples:

A) You have your data sitting in S3 and the user-specific data is in a browser-manageable area.

(E.g. this paper from Tableau research actually states dataset sizes that should fall into that category: https://research.tableau.com/sites/default/files/get_real.pd...)

In that scenario, you could eliminate your central server if your clients are smart enough.

B) You are talking about larger dataset sizes (GB) and want to explore them ad-hoc in your browser.

Uploading them is unrealistic and installing additional software is no longer ad-hoc enough.

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#15

Post by mytherin »

crimsoneer wrote: post #6

I'm still not sure I "get" the use case for DuckDB. From what I understand, it's like a nifty, in-memory SQL, but why is that better than just running PostGRES or Microsoft SQL server locally, where your data structures and tables and stuff have a lot more permanence? Like, my workflow is either I query an exiting remote corporate DB and do my initial data munging there, or get givne a data dump that I either work on…

DuckDB developer here. DuckDB is a regular RDBMS that has persistent ACID storage, but is tuned towards analytical workloads, i.e. read-heavy workloads with aggregates that require full scans of the data. Any data you write to tables is stored persistently on disk, and not all your data needs to fit in memory either.

Our tagline is “SQLite for analytics”, as DuckDB is an in-process database system similar to SQLite that is geared towards these types of workloads.

DuckDB has a flexible query engine, and also has support for directly running SQL queries (in parallel!) on top of Pandas [1] and Parquet [2] without requiring the data to be imported into the system.

[1] https://duckdb.org/2021/05/14/sql-on-pandas.html

[2] https://duckdb.org/2021/06/25/querying-parquet.html

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#16

Post by loxias »

crimsoneer wrote: post #6

I'm still not sure I "get" the use case for DuckDB. From what I understand, it's like a nifty, in-memory SQL, but why is that better than just running PostGRES or Microsoft SQL server locally, where your data structures and tables and stuff have a lot more permanence? Like, my workflow is either I query an exiting remote corporate DB and do my initial data munging there, or get givne a data dump that I either work on…

To me, the use case is really obvious: when you reached for SQLite but now want something with Moar Powah.

Now I've reduced it to a previously answered question: what's the use case for SQLite? ;) https://www.sqlite.org/whentouse.html

That being said, I don't see the point, and shudder at the idea of a web page's javascript doing anything which needs noticeable amounts of the CPU, but I'm a non-standard user...

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#18

Post by ankoh »

obeliskora wrote: post #17

There was neat post https://news.ycombinator.com/item?id=27016630 a while ago about about using sqlite on static pages with large datasets that wouldn't have to be loaded entirely. Does duckdb do something similar with arrow/parquet files or its own format?

Yes we do! DuckDB-Wasm can read files using HTTP range requests very similar to the sql.js-httpvfs from phiresky.

The blog post contains a few examples how this can be used, for example, to partially query Parquet files over the network.

E.g. just visit shell.duckdb.org and enter:

select * from 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parqu...' limit 10;

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#19

Post by 1egg0myegg0 »

crimsoneer wrote: post #6

I'm still not sure I "get" the use case for DuckDB. From what I understand, it's like a nifty, in-memory SQL, but why is that better than just running PostGRES or Microsoft SQL server locally, where your data structures and tables and stuff have a lot more permanence? Like, my workflow is either I query an exiting remote corporate DB and do my initial data munging there, or get givne a data dump that I either work on…

Check out this post for some comparisons with Pandas.

https://duckdb.org/2021/05/14/sql-on-pandas.html

DuckDB is often faster than Pandas, and it can handle larger than memory data. Plus, if you already know SQL, you don't have to become a Pandas expert to be productive in Python data munging. Pandas is still good, but now you can mix and match with SQL!

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#20

Post by 1egg0myegg0 »

ankoh wrote: post #18
obeliskora wrote: post #17

There was neat post https://news.ycombinator.com/item?id=27016630 a while ago about about using sqlite on static pages with large datasets that wouldn't have to be loaded entirely. Does duckdb do something similar with arrow/parquet files or its own format?

Yes we do! DuckDB-Wasm can read files using HTTP range requests very similar to the sql.js-httpvfs from phiresky. The blog post contains a few examples how this can be used, for example, to partially query Parquet files over the network. E.g. just visit shell.duckdb.org and enter: select * from ' https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parqu... ' limit 10;

NIIIICE! Data twitter was pretty excited about that cool SQLite trick - now you can turn it up a notch!
Post reply on HN