Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why DuckDB is my first choice for data processing

robinlinacre.com

Post reply on HN
Print view View original

101–110 of 124 posts

Re: Why DuckDB is my first choice for data processing

#101

Post by falconroar »

mrtimo wrote: post #19

What I love about duckdb: -- Support for .parquet, .json, .csv (note: Spotify listening history comes in a multiple .json files, something fun to play with). -- Support for glob reading, like: select * from 'tsa20*.csv' - so you can read hundreds of files (any type of file!) as if they were one file. -- if the files don't have the same schema, union_by_name is amazing. -- The .csv parser is amazing. Auto assigns type…

Polars has all of these benefits (to some degree), but also allows for larger-than-memory datasets.

Re: Why DuckDB is my first choice for data processing

#103

Post by falconroar »

anotherpaul wrote: post #92
steve_adams_86 wrote: post #41

It has become a favourite tool for me as well. I work with scientists who research BC's coastal environment, from airborne observation of glaciers to autonomous drones in the deep sea. We've got heaps of data. A while back I took a leap of faith with DuckDB as the data-processing engine for a new tool we're using to transform and validate biodiversity data. The goal is to take heaps of existing datasets and convert t…

What's the advantage over using Polars for the same task? It seems to me the natural competitor here and I vastly prefer the Polars syntax over SQL any day. So I was curious if I should try duckdb or stick with polars

Polars has all of the benefits of DuckDB (to some degree), but also allows for larger-than-memory datasets.

Re: Why DuckDB is my first choice for data processing

#104

Post by falconroar »

noo_u wrote: post #15

I'd say the author's thoughts are valid for basic data processing. Outside of that, most of claims in this article, such as: "We're moving towards a simpler world where most tabular data can be processed on a single large machine1 and the era of clusters is coming to an end for all but the largest datasets." become very debatable. Depending on how you want to pivot/ scale/augment your data, even datasets that seeming…

Polars also has all of these benefits (to some degree), but also allows for larger-than-memory datasets. Also has GPU backend, distributed backend, etc. Polars is heavily underrated, even with the recent hype.

Re: Why DuckDB is my first choice for data processing

#105

Post by steve_adams_86 »

falconroar wrote: post #103
anotherpaul wrote: post #92

Earlier quoted context omitted.

What's the advantage over using Polars for the same task? It seems to me the natural competitor here and I vastly prefer the Polars syntax over SQL any day. So I was curious if I should try duckdb or stick with polars

Polars has all of the benefits of DuckDB (to some degree), but also allows for larger-than-memory datasets.

DuckDB has this capability as well: https://duckdb.org/docs/stable/guides/performance/how_to_tun...

Re: Why DuckDB is my first choice for data processing

#106

Post by falconroar »

steve_adams_86 wrote: post #105
falconroar wrote: post #103

Earlier quoted context omitted.

Polars has all of the benefits of DuckDB (to some degree), but also allows for larger-than-memory datasets.

DuckDB has this capability as well: https://duckdb.org/docs/stable/guides/performance/how_to_tun...

Interesting, I wasn't aware; thanks for that. I will say, Polars' implementation is much more centered on out-of-core processing, and bypasses some of DuckDB's limitations ("DuckDB cannot yet offload some complex intermediate aggregate states to disk"). Both incredible pieces of software.

To expand on this, Polars' `LazyFrame` implementation allows for simple addition of new backends like GPU, streaming, and now distributed computing (though it's currently locked to a vendor). The DuckDB codebase just doesn't have this flexibility, though there are ways to get it to run on GPU using external software.

Re: Why DuckDB is my first choice for data processing

#107

Post by prometheon1 »

falconroar wrote: post #101
mrtimo wrote: post #19

What I love about duckdb: -- Support for .parquet, .json, .csv (note: Spotify listening history comes in a multiple .json files, something fun to play with). -- Support for glob reading, like: select * from 'tsa20*.csv' - so you can read hundreds of files (any type of file!) as if they were one file. -- if the files don't have the same schema, union_by_name is amazing. -- The .csv parser is amazing. Auto assigns type…

Polars has all of these benefits (to some degree), but also allows for larger-than-memory datasets.

DuckDB supports this as well, depending on which benchmark you look at it regularly performs better on those datasets than Polars.

Re: Why DuckDB is my first choice for data processing

#108

Post by geysersam »

steve_adams_86 wrote: post #41

It has become a favourite tool for me as well. I work with scientists who research BC's coastal environment, from airborne observation of glaciers to autonomous drones in the deep sea. We've got heaps of data. A while back I took a leap of faith with DuckDB as the data-processing engine for a new tool we're using to transform and validate biodiversity data. The goal is to take heaps of existing datasets and convert t…

What file formats are the existing datasets you have? I also work on data processing in a scientific domain where HDF5 is a common format. Unfortunately Duckdb doesn't support HDF5 out of the box, and the existing hdf5 extension wasn't fast enough and didn't have the features needed, so I made a new one based on the c++ extension template. I'd love to collaborate on it if anyone is interested.

Re: Why DuckDB is my first choice for data processing

#109

Post by camgunz »

falconroar wrote: post #104
noo_u wrote: post #15

I'd say the author's thoughts are valid for basic data processing. Outside of that, most of claims in this article, such as: "We're moving towards a simpler world where most tabular data can be processed on a single large machine1 and the era of clusters is coming to an end for all but the largest datasets." become very debatable. Depending on how you want to pivot/ scale/augment your data, even datasets that seeming…

Polars also has all of these benefits (to some degree), but also allows for larger-than-memory datasets. Also has GPU backend, distributed backend, etc. Polars is heavily underrated, even with the recent hype.

I downvoted all your recs for polars, 1 because this is a DuckDB thread and it's low-key rude, and 2 because there are 4 of them. I wouldn't have minded if there were a single post that were like "DuckDB is cool, polars could be an alternative if..."

Re: Why DuckDB is my first choice for data processing

#110

Post by noworriesnate »

falconroar wrote: post #106
steve_adams_86 wrote: post #105

Earlier quoted context omitted.

DuckDB has this capability as well: https://duckdb.org/docs/stable/guides/performance/how_to_tun...

Interesting, I wasn't aware; thanks for that. I will say, Polars' implementation is much more centered on out-of-core processing, and bypasses some of DuckDB's limitations ("DuckDB cannot yet offload some complex intermediate aggregate states to disk"). Both incredible pieces of software. To expand on this, Polars' `LazyFrame` implementation allows for simple addition of new backends like GPU, streaming, and now dist…

Have you seen Ibis[1]? It's a dataframe API that translates calls to it into various backends, including Polars and DuckDB. I've messed around with it a little for cases where data engineering transforms had to use pyspark but I wanted to do exploratory analysis in an environment that didn't have pyspark.

[1] https://ibis-project.org/

Post reply on HN