Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Preview of DuckDB v2.0

duckdb.org

Post reply on HN
Print view View original

71–80 of 144 posts

Re: A Preview of DuckDB v2.0

#71

Post by tpetry »

amluto wrote: post #26

If I could have a pet feature added to DuckDB, it would be some form of native ordered table. In a database like Clickhouse or any of the dedicated time series DBMSes or log stores, there’s a built-in concept that a table might have an order, and the database will optimize based on the order. But, for databases that are logically just bags of rows (traditional DBMSes and also DuckDB [0]), you either need an index or…

You‘re not the only one interested in this. But seems its a big change that would have to change many parts within DuckdB:

https://github.com/duckdb/duckdb/discussions/8444

Re: A Preview of DuckDB v2.0

#73

Post by onnimonni »

rrgok wrote: post #70

Sometimes I prefer DuckDB query language to MySQL or Postgres. It is not even about performance, it just feels right. I just connect to my DBS from DuckDB just to use its query language. And besides it gives you a single language frontend.

This! For example I tried to ask PosgreSQL to Add support for QUALIFY keyword since it makes window queries much more simpler to use. Postgres didnt want it. DuckDB maintainers are always thinking about the DX and all nice syntactic sugar.

Re: A Preview of DuckDB v2.0

#74

Post by nikolay »

It's sad that almost no migration framework supports DuckDB, and overall support is highly limited, but it's a great product I've been using daily for 6 months without encountering a single issue. I hope v2 brings DuckDB more attention and increases third-party support!

Re: A Preview of DuckDB v2.0

#75

Post by amluto »

tpetry wrote: post #71
amluto wrote: post #26

If I could have a pet feature added to DuckDB, it would be some form of native ordered table. In a database like Clickhouse or any of the dedicated time series DBMSes or log stores, there’s a built-in concept that a table might have an order, and the database will optimize based on the order. But, for databases that are logically just bags of rows (traditional DBMSes and also DuckDB [0]), you either need an index or…

You‘re not the only one interested in this. But seems its a big change that would have to change many parts within DuckdB: https://github.com/duckdb/duckdb/discussions/8444

Neat!

I’m sort of amazing that nothing in that thread mentioned ClickHouse, even just as a product that has this capability and from which one might take syntactical inspiration.

Re: A Preview of DuckDB v2.0

#77

Post by hans_castorp »

onnimonni wrote: post #73
rrgok wrote: post #70

Sometimes I prefer DuckDB query language to MySQL or Postgres. It is not even about performance, it just feels right. I just connect to my DBS from DuckDB just to use its query language. And besides it gives you a single language frontend.

This! For example I tried to ask PosgreSQL to Add support for QUALIFY keyword since it makes window queries much more simpler to use. Postgres didnt want it. DuckDB maintainers are always thinking about the DX and all nice syntactic sugar.

The Postgres devs rejected it, because it wasn't part of the standard back then, and apparently work was going on, to include it into the standard.

Now that it's included in the standard, there is work to implement it:

https://hackorum.dev/topics/51960#message-1125429

Re: A Preview of DuckDB v2.0

#79

Post by biophysboy »

dartharva wrote: post #54

Would be really cool if they were to add statistical functions too. I'd jump at the chance of getting to use this over pandas

You might know this already, but you can query pandas/polars/arrow tables directly w/ duckdb and use whatever stats packages you feel like alongside it in the same python script. I feel like they do a decent job sticking to the simpler statistical fans that make sense in sql.

Re: A Preview of DuckDB v2.0

#80

Post by jtbaker »

staticautomatic wrote: post #65
jtbaker wrote: post #25

Earlier quoted context omitted.

I've got a couple of different use cases: - ETL pipelines running on K8s nodes. Using their streaming processing engine means I can run smaller pods/nodes if needed, for datasets that may have required large dataframe-like transformations that may have buffered a big dataset into memory previously. - A CLI distributed to an internal team to do a postprocessing step on a large modeling dataset - to get it into a consu…

Similar here. Lots of places where we replaced Pandas with DuckDB for transformations. Also have scriptable custom dashboards running on top of BigQuery data pre-aggregated and extracted to parquet on GCS. It's way faster and the only limiting factor is your viz library. It was pretty easy to build and the only big gotcha I encountered was finding, somewhat counter-intuitively, that it's often best minimize partition…

> the only big gotcha I encountered was finding, somewhat counter-intuitively, that it's often best minimize partitioning.

For parquet, I think with partitioning, it's really important to be mindful of the ordering of the data within the parquet file and also the query patterns of the main use cases. A little hard to generalize well to every pattern I guess.

Post reply on HN