11–20 of 40 posts
uhm why would you ever use this instead of sqlite
DuckDB is optimised for analytical tasks, whereas SQLite isn't.
I know I'm repeating my self (it must be my third comment on HN about this topic), but this does not match my experience at all. DuckDB will error-out with an out-of-memory exception in very simple DISTINCT ON / GROUP BY queries. Even with a temporary file, an on-disk database and not keeping the initial order. On any version of DuckDB.
While I am a Linux user, tried this on a available Windows 10 machine (I know...Yuck!)
1) Setup and install no prob. Tracked the duckdb process memory usage with PowerShell, like this:
Get-Process -Name duckdb | Select-Object Name, @{Name="Memory (MB)";Expression={[math]::round($_.WorkingSet64/1MB,2)}}
Section in the blog: "FIRST ANALYTICS PROJECT"
Table has 7100 rows as seen like this:
SELECT COUNT(*) AS RowCount FROM netflix;
The one below, being one example:
SELECT DISTINCT ON (Title)
Title,
"As of",
Rank
FROM netflix
ORDER BY Title, "As of" DESC;
I also tested, with the parquet file from the thread mentioned below by mgt19937. It is 89475 rows. Did some complex DISTINCT ON / GROUP BY on it, without seeing neither explosive memory use or something similar to a memory leak.
Do you have a more specific example?
Earlier quoted context omitted.
DuckDB is optimised for analytical tasks, whereas SQLite isn't.
For those not aware: analytical tasks involve a lot of groupings, aggregations, running sums/averages, sorting, ranking etc. Columnar databases like duckdb are more focused on those tasks so you can do these tasks much faster.
Really, is this what's getting praised? I mean specifically the first point: the whole "just paste the url into the DB" - thing, + inferring the column names. That looks like the laziest and shakiest basis, and if I ever saw that in production i d be both stunned and scared
Really, is this what's getting praised? I mean specifically the first point: the whole "just paste the url into the DB" - thing, + inferring the column names. That looks like the laziest and shakiest basis, and if I ever saw that in production i d be both stunned and scared
csv metadata inference in duckdb is amazing. There is some research in this domain and duckdb does a great job there, but yes there might be some really strange csv files, which require manual intervention.
Earlier quoted context omitted.
For those not aware: analytical tasks involve a lot of groupings, aggregations, running sums/averages, sorting, ranking etc. Columnar databases like duckdb are more focused on those tasks so you can do these tasks much faster.
Ty! Never got into the analytics side of things
Or the advantage of columnar file formats, like ORC or Parquet, for analytical queries. Normally you are only interested in a few columns.
I know I'm repeating my self (it must be my third comment on HN about this topic), but this does not match my experience at all. DuckDB will error-out with an out-of-memory exception in very simple DISTINCT ON / GROUP BY queries. Even with a temporary file, an on-disk database and not keeping the initial order. On any version of DuckDB.
I know I'm repeating my self (it must be my third comment on HN about this topic), but this does not match my experience at all. DuckDB will error-out with an out-of-memory exception in very simple DISTINCT ON / GROUP BY queries. Even with a temporary file, an on-disk database and not keeping the initial order. On any version of DuckDB.
DuckDB has great ergonomics for moving data between different databases and making copies for local analysis. The one thing that differed in my experience with it from the author’s is how much of the Postgres sql dialect (and extensions) it supports. Attempting to run my Postgres analytics sql code in duckdb errors out on most json operations - to be fair, the DuckDB json functions have cleaner names than jsonb_path_…
If I understand it correctly, when you use it it:
- Pulls the minimal data required (inferred from the query) from postgres into duckdb
- Executes your query using duckdb execution engine
BUT, if your postgres function is not supported by DuckDB I think you can use the `postgres_execute` [2] to execute the function within postgres itself
I'm not sure whether you can e.g do a CTE pipeline that starts with postgres_execute, and then executes Duckdb sql in later stages of the pipeline
[1] https://duckdb.org/docs/extensions/postgres.html#running-sql... [2]https://duckdb.org/docs/extensions/postgres.html#the-postgre...