111–120 of 124 posts
stupid question: - what is wrong with postgresql for doing this?
What I love about duckdb: -- Support for .parquet, .json, .csv (note: Spotify listening history comes in a multiple .json files, something fun to play with). -- Support for glob reading, like: select * from 'tsa20*.csv' - so you can read hundreds of files (any type of file!) as if they were one file. -- if the files don't have the same schema, union_by_name is amazing. -- The .csv parser is amazing. Auto assigns type…
I love duckdb, I use it as much as I can. I just wish that the support for node/bun was as good as python. And I wish that they would bundle it differently for node/bun - the way it is now it depends on a dynamic link to a library which means I cant bundle it into a bun executable.
stupid question: - what is wrong with postgresql for doing this?
postgres (without extensions) is slower for analytical queries since it stores data as rows, not columns. Also, duckdb is an in-memory database so it's more comparable to sqlite than postgres.
Been quite a fan of DuckDB and we actually even use it in production. But coincidentally today I was exploring memory usage and I believe I'm finding memory leaks. Anybody have similar experiences? Still debugging more deeply but looking reasonably conclusive atm.
As far as I can tell pretty conclusive results: https://github.com/duckdb/duckdb/issues/20569 If someone can explain this or has a fix for it I'd love to hear it!
DuckDB has a memory_limit setting with a default of 80% of RAM. If you want to set a lower limit you can do something like
SET memory_limit = '1GB';
I dont get the either/or posts here about polars. Some times I do duckdb sql queries against a polars df, some times I convert a duckdb result to polars. Its great to use whatever is best for the situation.
That said, polars is good, and if the team agree to standardise on it then that's a totally reasonable choice.
I guess one of my reservations is I've been (historically) burned by decisions within data eng teams to use pandas, causing all sorts of problems with data typing and memory and eventually having to rewrite it all. But I accept polars doesn't suffer from the same problems (and actually some of them are even mitigated in more recent versions of pandas)
Earlier quoted context omitted.
DuckDB has this capability as well: https://duckdb.org/docs/stable/guides/performance/how_to_tun...
Interesting, I wasn't aware; thanks for that. I will say, Polars' implementation is much more centered on out-of-core processing, and bypasses some of DuckDB's limitations ("DuckDB cannot yet offload some complex intermediate aggregate states to disk"). Both incredible pieces of software. To expand on this, Polars' `LazyFrame` implementation allows for simple addition of new backends like GPU, streaming, and now dist…
Being able to use distributed backends to process frames sounds kind of incredible, but I can't imagine my little projects ever making use of it. Still, very cool stuff.