Why DuckDB is my first choice for data processing • Hacker News Board

Re: Why DuckDB is my first choice for data processing

#121

Post by steve_adams_86 » Mon, Jan 19, 2026, 4:48 AM UTC

geysersam wrote: post #108

steve_adams_86 wrote: post #41

It has become a favourite tool for me as well. I work with scientists who research BC's coastal environment, from airborne observation of glaciers to autonomous drones in the deep sea. We've got heaps of data. A while back I took a leap of faith with DuckDB as the data-processing engine for a new tool we're using to transform and validate biodiversity data. The goal is to take heaps of existing datasets and convert t…

What file formats are the existing datasets you have? I also work on data processing in a scientific domain where HDF5 is a common format. Unfortunately Duckdb doesn't support HDF5 out of the box, and the existing hdf5 extension wasn't fast enough and didn't have the features needed, so I made a new one based on the c++ extension template. I'd love to collaborate on it if anyone is interested.

That's really fascinating. Is your format open source? I don't know if I'd have overlapping needs for something like that (though I did investigate hdf5 early on, it seemed very promising as a place to store our outputs) but I'd be curious to explore it and see what you're doing with it.

Right now we typically read from CSV or Excel, because that's what the scientists prefer to work with. For better or worse. There's a bit of parquet kicking around. The wrappers around handling imports for DuckDB are very, very thin. It handles just about everything seamlessly

Is this the extension that was too slow? https://duckdb.org/community_extensions/extensions/hdf5

Re: Why DuckDB is my first choice for data processing

#122

Post by exagolo » Mon, Jan 19, 2026, 10:23 AM UTC

wood_spirit wrote: post #52

theLiminator wrote: post #51

Earlier quoted context omitted.

Yeah, i'm also similarly confused. > "SQL should be the first option considered for new data engineering work. It’s robust, fast, future-proof and testable. With a bit of care, it’s clear and readable." (over polars/pandas etc) SQL has nothing to do with fast. Not sure what makes it any more testable than polars? Future-proof in what way? I guess they mean your SQL dialect won't have breaking changes?

I’m also a duckdb convert. All my notebooks have moved from Pandas and polars to Duckdb. It is faster to write and faster to read (after you return to a notebook after time away) and often faster to run. Certainly not slower to run. My current habit is to suck down big datasets to parquet shards and then just query them with a wildcard in duckdb. I move to bigquery when doing true “big data” but a few GB of extract f…

For the bigger tasks, Exasol might also be a very neat option for you. We have a free personal edition that can scale regarding data volumes, #servers (MPP architecture) and complex workloads.

Recently, we have also compared ourselves against DuckDB and were 4 times faster even on a single node. We are in-memory optimized, but data doesn't need to fit in the RAM.

Disclaimer: I'm CTO@Exasol

Re: Why DuckDB is my first choice for data processing

#123

Post by morgul » Fri, Jan 23, 2026, 3:08 PM UTC

I did large-scale molecular data engineering with DuckDB and Polars (https://github.com/scikit-fingerprints/MolPILE_dataset, https://arxiv.org/abs/2509.18353 for those interested). Both were amazing, but for large-scale JOINs, only DuckDB didn't result in OOM. Really a pleasure to use too.

I could also JOIN local CSV datasets, Postgres database, and even Excel files from chemists. All of this in Jupyter Notebook and really seamless Python integration. This also means that one can easily do heavy lifting in DuckDB, export to Polars variable, put into Plotly, and get an interactive plot. Neat stuff.

Re: Why DuckDB is my first choice for data processing

#124

Post by morgul » Fri, Jan 23, 2026, 3:09 PM UTC

I did large-scale molecular data engineering with DuckDB and Polars (https://github.com/scikit-fingerprints/MolPILE_dataset, https://arxiv.org/abs/2509.18353 for those interested). Both were amazing, but for large-scale JOINs, only DuckDB didn't result in OOM (tried Dask, DuckDB, Polars). Really a pleasure to use too.

I could also JOIN local CSV datasets, Postgres database, and even Excel files from chemists. All of this in Jupyter Notebook and really seamless Python integration. This also means that one can easily do heavy lifting in DuckDB, export to Polars variable, put into Plotly, and get an interactive plot. Neat stuff.