Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why DuckDB is my first choice for data processing

robinlinacre.com

Post reply on HN
Print view View original

51–60 of 124 posts

Re: Why DuckDB is my first choice for data processing

#51

Post by theLiminator »

noo_u wrote: post #15

I'd say the author's thoughts are valid for basic data processing. Outside of that, most of claims in this article, such as: "We're moving towards a simpler world where most tabular data can be processed on a single large machine1 and the era of clusters is coming to an end for all but the largest datasets." become very debatable. Depending on how you want to pivot/ scale/augment your data, even datasets that seeming…

Yeah, i'm also similarly confused.

> "SQL should be the first option considered for new data engineering work. It’s robust, fast, future-proof and testable. With a bit of care, it’s clear and readable." (over polars/pandas etc)

SQL has nothing to do with fast. Not sure what makes it any more testable than polars? Future-proof in what way? I guess they mean your SQL dialect won't have breaking changes?

Re: Why DuckDB is my first choice for data processing

#52

Post by wood_spirit »

theLiminator wrote: post #51
noo_u wrote: post #15

I'd say the author's thoughts are valid for basic data processing. Outside of that, most of claims in this article, such as: "We're moving towards a simpler world where most tabular data can be processed on a single large machine1 and the era of clusters is coming to an end for all but the largest datasets." become very debatable. Depending on how you want to pivot/ scale/augment your data, even datasets that seeming…

Yeah, i'm also similarly confused. > "SQL should be the first option considered for new data engineering work. It’s robust, fast, future-proof and testable. With a bit of care, it’s clear and readable." (over polars/pandas etc) SQL has nothing to do with fast. Not sure what makes it any more testable than polars? Future-proof in what way? I guess they mean your SQL dialect won't have breaking changes?

I’m also a duckdb convert. All my notebooks have moved from Pandas and polars to Duckdb. It is faster to write and faster to read (after you return to a notebook after time away) and often faster to run. Certainly not slower to run.

My current habit is to suck down big datasets to parquet shards and then just query them with a wildcard in duckdb. I move to bigquery when doing true “big data” but a few GB of extract from BQ to a notebook VM disk and duckdb is super ergonomic and performant most of the time.

It’s the sql that I like. Being a veteran of when the world went mad for nosql it is just so nice to experience the revenge of sql.

Re: Why DuckDB is my first choice for data processing

#53

Post by 0cf8612b2e1e »

tjchear wrote: post #7

I’ve not used duckdb before nor do I do much data analysis so I am curious about this one aspect of processing medium sized json/csv with it: the data are not indexed, so any non-trivial query would require a full scan. Is duckdb so fast that this is never really a problem for most folks?

Visidata is in Python and has offered “real time” analytics of fixed files for a long time. Computers are stupidly fast. You can do a lot of operations within a few seconds time window.

Re: Why DuckDB is my first choice for data processing

#54

Post by theLiminator »

wood_spirit wrote: post #52
theLiminator wrote: post #51

Earlier quoted context omitted.

Yeah, i'm also similarly confused. > "SQL should be the first option considered for new data engineering work. It’s robust, fast, future-proof and testable. With a bit of care, it’s clear and readable." (over polars/pandas etc) SQL has nothing to do with fast. Not sure what makes it any more testable than polars? Future-proof in what way? I guess they mean your SQL dialect won't have breaking changes?

I’m also a duckdb convert. All my notebooks have moved from Pandas and polars to Duckdb. It is faster to write and faster to read (after you return to a notebook after time away) and often faster to run. Certainly not slower to run. My current habit is to suck down big datasets to parquet shards and then just query them with a wildcard in duckdb. I move to bigquery when doing true “big data” but a few GB of extract f…

I personally find polars easier to read/write than sql. Especially when you start doing UDFs with numpy/et. al. I think for me, duckdb's clear edge is the cli experience.

> It is faster to write and faster to read

At least on clickbench, polars and duckdb are roughly comparable (with polars edging out duckdb).

Re: Why DuckDB is my first choice for data processing

#56

Post by yakkomajuri »

Been quite a fan of DuckDB and we actually even use it in production.

But coincidentally today I was exploring memory usage and I believe I'm finding memory leaks. Anybody have similar experiences?

Still debugging more deeply but looking reasonably conclusive atm.

Re: Why DuckDB is my first choice for data processing

#57

Post by davidtm »

It is the darling of laptop-based analysis. I’m always surprised that duckdb doesn’t get used more to enhance existing, and stubbornly bedded-in data pipelines. We use it in browser and on server at Count (https://count.co) to make data warehouses more responsive for multiuser and agentic work.

Re: Why DuckDB is my first choice for data processing

#58

Post by exographicskip »

mrtimo wrote: post #19

What I love about duckdb: -- Support for .parquet, .json, .csv (note: Spotify listening history comes in a multiple .json files, something fun to play with). -- Support for glob reading, like: select * from 'tsa20*.csv' - so you can read hundreds of files (any type of file!) as if they were one file. -- if the files don't have the same schema, union_by_name is amazing. -- The .csv parser is amazing. Auto assigns type…

Thanks for the excellent comment! Now excuse me while I go export my spotify history to play around with duckdb <3

Re: Why DuckDB is my first choice for data processing

#59

Post by [deleted] »

theLiminator wrote: post #54
wood_spirit wrote: post #52

Earlier quoted context omitted.

I’m also a duckdb convert. All my notebooks have moved from Pandas and polars to Duckdb. It is faster to write and faster to read (after you return to a notebook after time away) and often faster to run. Certainly not slower to run. My current habit is to suck down big datasets to parquet shards and then just query them with a wildcard in duckdb. I move to bigquery when doing true “big data” but a few GB of extract f…

I personally find polars easier to read/write than sql. Especially when you start doing UDFs with numpy/et. al. I think for me, duckdb's clear edge is the cli experience. > It is faster to write and faster to read At least on clickbench, polars and duckdb are roughly comparable (with polars edging out duckdb).

[deleted]

Re: Why DuckDB is my first choice for data processing

#60

Post by PLenz »

Duckdb is wonderful. I have several multi-TB pipelines that I moved over from spark and dask. It's so much easier to think in terms of knew machine that gets it's resources used efficiently instead of distributed/cloud processes. It even let me take some big data things back to on-prem from aws.
Post reply on HN