Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB-Wasm: Efficient analytical SQL in the browser

duckdb.org

Post reply on HN
Print view View original

21–30 of 62 posts

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#21

Post by obeliskora »

ankoh wrote: post #18
obeliskora wrote: post #17

There was neat post https://news.ycombinator.com/item?id=27016630 a while ago about about using sqlite on static pages with large datasets that wouldn't have to be loaded entirely. Does duckdb do something similar with arrow/parquet files or its own format?

Yes we do! DuckDB-Wasm can read files using HTTP range requests very similar to the sql.js-httpvfs from phiresky. The blog post contains a few examples how this can be used, for example, to partially query Parquet files over the network. E.g. just visit shell.duckdb.org and enter: select * from ' https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parqu... ' limit 10;

That's really neat! Can you control the cache too?

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#22

Post by jamesrr39 »

catawbasam wrote: post #7
crimsoneer wrote: post #6

I'm still not sure I "get" the use case for DuckDB. From what I understand, it's like a nifty, in-memory SQL, but why is that better than just running PostGRES or Microsoft SQL server locally, where your data structures and tables and stuff have a lot more permanence? Like, my workflow is either I query an exiting remote corporate DB and do my initial data munging there, or get givne a data dump that I either work on…

Not just in-memory. It's pretty convenient if you have a set of Parquet files with common schema. Fairly snappy and doesn't have to fit in memory.

I'm using duckdb for querying parquet files as well. It's an awesome tool, so nice to just "look into" parquet files with SQL.

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#23

Post by ankoh »

obeliskora wrote: post #21
ankoh wrote: post #18

Earlier quoted context omitted.

Yes we do! DuckDB-Wasm can read files using HTTP range requests very similar to the sql.js-httpvfs from phiresky. The blog post contains a few examples how this can be used, for example, to partially query Parquet files over the network. E.g. just visit shell.duckdb.org and enter: select * from ' https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parqu... ' limit 10;

That's really neat! Can you control the cache too?

DuckDB-Wasm uses a traditional buffer manager and evicts pages using a combination of FIFO + LRU (to distinguish sequential scans from hot pages like the Parquet metadata).

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#24

Post by Dayshine »

crimsoneer wrote: post #6

I'm still not sure I "get" the use case for DuckDB. From what I understand, it's like a nifty, in-memory SQL, but why is that better than just running PostGRES or Microsoft SQL server locally, where your data structures and tables and stuff have a lot more permanence? Like, my workflow is either I query an exiting remote corporate DB and do my initial data munging there, or get givne a data dump that I either work on…

The simple answer noone else seems to have mentioned: SQLLite has quite a low limit on the number of columns it supports, which is a problem for data analytics which often prefers wide over long.

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#25

Post by brandmeyer »

mytherin wrote: post #15
crimsoneer wrote: post #6

I'm still not sure I "get" the use case for DuckDB. From what I understand, it's like a nifty, in-memory SQL, but why is that better than just running PostGRES or Microsoft SQL server locally, where your data structures and tables and stuff have a lot more permanence? Like, my workflow is either I query an exiting remote corporate DB and do my initial data munging there, or get givne a data dump that I either work on…

DuckDB developer here. DuckDB is a regular RDBMS that has persistent ACID storage, but is tuned towards analytical workloads, i.e. read-heavy workloads with aggregates that require full scans of the data. Any data you write to tables is stored persistently on disk, and not all your data needs to fit in memory either. Our tagline is “SQLite for analytics”, as DuckDB is an in-process database system similar to SQLite t…

Maybe this is a silly question: Why is the A/B choice between a row-major database and a column-major database, instead of between row-major tables and column-major tables within a flexible database?

What's stopping the other leading brands from implementing columnar storage, queries, and such with a COLUMN MAJOR table attribute?

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#27

Post by mytherin »

brandmeyer wrote: post #25
mytherin wrote: post #15

Earlier quoted context omitted.

DuckDB developer here. DuckDB is a regular RDBMS that has persistent ACID storage, but is tuned towards analytical workloads, i.e. read-heavy workloads with aggregates that require full scans of the data. Any data you write to tables is stored persistently on disk, and not all your data needs to fit in memory either. Our tagline is “SQLite for analytics”, as DuckDB is an in-process database system similar to SQLite t…

Maybe this is a silly question: Why is the A/B choice between a row-major database and a column-major database, instead of between row-major tables and column-major tables within a flexible database? What's stopping the other leading brands from implementing columnar storage, queries, and such with a COLUMN MAJOR table attribute?

Some databases do offer both, but it is much more involved than just changing the storage model. The entire query execution model needs to adapt to columnar execution. You can simulate a column store model in a row-store database by splitting a table into a series of single-column tables, but the performance benefits you will capture are much smaller than a system that is designed and optimized for column store execution.

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#28

Post by deshpand »

jamesrr39 wrote: post #22
catawbasam wrote: post #7

Earlier quoted context omitted.

Not just in-memory. It's pretty convenient if you have a set of Parquet files with common schema. Fairly snappy and doesn't have to fit in memory.

I'm using duckdb for querying parquet files as well. It's an awesome tool, so nice to just "look into" parquet files with SQL.

Many enterprises are coming up with patterns where they replicate the data from the database (say Redshift) into parquet files (data lake?) and directing more traffic including analytical workloads onto the parquet files.

duckdb will be very useful here, instead of having to use Redshift Spectrum or whatever.

Re: DuckDB-Wasm: Efficient analytical SQL in the browser

#29

Post by pantsforbirds »

Anyone have a good benchmark comparing DuckDb to Parquet/Avro/ORC etc.? Super curious to see how some of those workflows might compare. Obviously at scale its going to be different, but using a single parquet file/dataset as a db replacement isn't an uncommon thing in DS/ML work.
Post reply on HN