31–40 of 62 posts
I’ve been thinking of using absurd-sql for it since I saw https://news.ycombinator.com/item?id=28156831 last week
Anyone have a good benchmark comparing DuckDb to Parquet/Avro/ORC etc.? Super curious to see how some of those workflows might compare. Obviously at scale its going to be different, but using a single parquet file/dataset as a db replacement isn't an uncommon thing in DS/ML work.
Interesting.. Would this be effective at loading a remote CSV file with a million rows, then performing basic GROUP BY COUNTs on it so I can render bar charts? I’ve been thinking of using absurd-sql for it since I saw https://news.ycombinator.com/item?id=28156831 last week
Querying CSV files is particularly painful over the network since we still have to read everything for a full scan.
With Parquet, you would at least only have to read the columns of group by keys and aggregate arguments.
Try it out and share your experiences with us!
I'm still not sure I "get" the use case for DuckDB. From what I understand, it's like a nifty, in-memory SQL, but why is that better than just running PostGRES or Microsoft SQL server locally, where your data structures and tables and stuff have a lot more permanence? Like, my workflow is either I query an exiting remote corporate DB and do my initial data munging there, or get givne a data dump that I either work on…
And we may not want to upload the data into postgres or another database. We can just work with parquet files and run in-process queries.
Interesting.. Would this be effective at loading a remote CSV file with a million rows, then performing basic GROUP BY COUNTs on it so I can render bar charts? I’ve been thinking of using absurd-sql for it since I saw https://news.ycombinator.com/item?id=28156831 last week
Here's 1,000,000 rows example I just threw together for you
https://bl.ocks.org/texodus/3802a8671fa77399c7842fd0deffe925
and a CSV example, you try yours right now
https://bl.ocks.org/texodus/02d8fd10aef21b19d6165cf92e43e668
Gonna throw this out there: main thing I'm looking for from an embedded DB is better on disk compression; I've been toying with RocksDB, but it's hard to tune optimally & it's really too low level for my needs.
[1] > ipython
import numpy as np
duckdb = [0.855, 0.179, 0.151, 0.197, 0.086, 0.319, 0.236, 0.351, 0.276, 0.194, 0.086, 0.137, 0.377]
sqlite = [8.441, 1.758, 0.384, 1.965, 1.294, 2.677, 4.126, 1.238, 1.080, 5.887, 1.194, 0.453, 1.272]
print((np.quantile(sqlite, q=[0.1, 0.9]) / np.quantile(duckdb, q=[0.1, 0.9])).round())
There was neat post https://news.ycombinator.com/item?id=27016630 a while ago about about using sqlite on static pages with large datasets that wouldn't have to be loaded entirely. Does duckdb do something similar with arrow/parquet files or its own format?
Yes we do! DuckDB-Wasm can read files using HTTP range requests very similar to the sql.js-httpvfs from phiresky. The blog post contains a few examples how this can be used, for example, to partially query Parquet files over the network. E.g. just visit shell.duckdb.org and enter: select * from ' https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parqu... ' limit 10;
I'm still not sure I "get" the use case for DuckDB. From what I understand, it's like a nifty, in-memory SQL, but why is that better than just running PostGRES or Microsoft SQL server locally, where your data structures and tables and stuff have a lot more permanence? Like, my workflow is either I query an exiting remote corporate DB and do my initial data munging there, or get givne a data dump that I either work on…
DuckDB is columnar, so in theory a lot faster than Postgres or SQL server for Analytical workloads. DuckDB is to Clickhouse, TimescaleDB, Redshift, etc as SQLite is to Postgres, MySQL, SQL Server.
Anyone have a good benchmark comparing DuckDb to Parquet/Avro/ORC etc.? Super curious to see how some of those workflows might compare. Obviously at scale its going to be different, but using a single parquet file/dataset as a db replacement isn't an uncommon thing in DS/ML work.
Why compare DuckDB to Parquet when you can use DuckDB and Parquet [1] :) [1] https://duckdb.org/2021/06/25/querying-parquet.html