Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB Internals Part 1

greybeam.ai

Post reply on HN
Print view View original

161–165 of 165 posts

Re: DuckDB Internals Part 1

#161

Post by refactor_master »

brikym wrote: post #45
refactor_master wrote: post #39

Earlier quoted context omitted.

The better question is, why is DuckDB so popular when one can use Polars which has a sane, lintable, typesafe API compared to the mess that is SQL: WITH lagged AS ( SELECT *, LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_time FROM events ), sessions AS ( SELECT *, SUM(COALESCE((date_diff('minute', prev_time, event_time) > 30)::INT, 1)) OVER (PARTITION BY user_id ORDER BY event_time) AS sessi…

Polars typesafe? It doesn't show you any errors until runtime right? Kusto query language is the best I've seen at type safety and I wish open source DBs would steal some ideas from it.

Yes, the most of its safety comes after you compile the graph. In that sense, it's "compile-safe" strictly speaking, which puts it on par with DuckDBs validation step. But you don't need to load any data to validate the execution graph (as opposed to Pandas).

Re: DuckDB Internals Part 1

#162

Post by tannenfreund87 »

bdcravens wrote: post #8
jdw64 wrote: post #6

The data scientists I work with use this. Why do they use it? I don't really know much about it, but I've noticed they use it quite often. I mainly use MySQL or PostgreSQL. What are the advantages of DuckDB? It seems like they usually use it as an alternative to Pandas.

Primarily the ability to work directly with data in its native format (CSV for example) without needing ETL.

DuckDB is a column-oriented database. This is why it's so fast compared to SQLite for a lot of analytical queries. CSV is a row-based format. DuckDB has to load it completley to memory before it can run any queries. Depending on the amount of RAM and Swap you have avialable, it's better to use parquet or similar column-based formats for data storage. Even SQLite as format is faster than CSV, since SQLite is already optimized, integers and floats take less space in SQLite format than as CSV.

Re: DuckDB Internals Part 1

#163

Post by tannenfreund87 »

holografix wrote: post #24

Why is DuckDB so popular when one can use Python + Pandas? Better perf + SQL is that mostly it?

Because I can just write SQL to access all my data. The data comes from CSVs, SQLite, MySQL, Postgres, ...

I can write SQL everywhere. I write SQL from within my R scripts, I write SQL from within my Python scripts. SQL is THE language for querying databases, others came and went, SQL will stay with us for the next decades. Even the dbplyr people say (parapfhrased): If you can't express what you want with the tidyverse, just write your SQL and load the result as a dataframe.

Re: DuckDB Internals Part 1

#165

Post by melih1im »

smithclay wrote: post #9

If you're reading this and curious: consider writing a duckdb community extension* or contributing to an existing one* duckdb is becoming a kind of data superglue between a lot of data ecosystems (GIS, observability, analytics, lakehouses, object storage, etc) that don't talk to each other typically, and it's worth checking out in 2026. * https://github.com/duckdb/extension-template * https://duckdb.org/community_ext…

DuckDB is amazing, especially extensions.
Post reply on HN