Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Preview of DuckDB v2.0

duckdb.org

Post reply on HN
Print view View original

21–30 of 144 posts

Re: A Preview of DuckDB v2.0

#21

Post by ericpauley »

fifilura wrote: post #20
jtbaker wrote: post #8

DuckDB is one of the things I've been most excited about in a long time. Introduced it to projects at 3 companies since 2023, greatly lowering resource requirements and running it in a variety of environments. Just having the ability to do out of core bigger than memory data processing on lower end consumer grade hardware is remarkable. Thanks to the team for everything!

My favourite is AWS Athena (backed by Trino). "If we use this we get indefinite RAM indefinite CPU and do not need to host a server". I had an impression that DuckDB was not great at distributing work to other machines, but good at doing it locally? Am I wrong?

Athena + Clickhouse has been an absolute game changer for us. Perfect combo for OLAP + deeper filtering that we can’t necessarily pre-index for.

Re: A Preview of DuckDB v2.0

#22

Post by badatnames »

Looks like an awesome release, but the smell of AI from that post is horrid.

Here is a wild idea: is it really so hard to edit out sentences structured and punctuated like this - it's so painfully obvious and distracts from the content. The effect is real.

Re: A Preview of DuckDB v2.0

#25

Post by jtbaker »

throwaw12 wrote: post #9
jtbaker wrote: post #8

DuckDB is one of the things I've been most excited about in a long time. Introduced it to projects at 3 companies since 2023, greatly lowering resource requirements and running it in a variety of environments. Just having the ability to do out of core bigger than memory data processing on lower end consumer grade hardware is remarkable. Thanks to the team for everything!

Curious to learn more about how people are using it? Are they downloading parquet files and running analyses locally, or are they connecting to Iceberg-like data lake and leveraging DuckDBs query engine capabilities or have you exposed an interface (REST, UI) to query your data?

I've got a couple of different use cases:

- ETL pipelines running on K8s nodes. Using their streaming processing engine means I can run smaller pods/nodes if needed, for datasets that may have required large dataframe-like transformations that may have buffered a big dataset into memory previously.

- A CLI distributed to an internal team to do a postprocessing step on a large modeling dataset - to get it into a consumable format and upload it to a bucket as a .db file.

- A SvelteKit app that used the node duckdb bindings to attach to the .db on the bucket and explore the results through a suite of BI tools. These tables have millions of rows, and would be pretty heavy to store in PG. The DuckDB version works really, really well.

Re: A Preview of DuckDB v2.0

#26

Post by amluto »

If I could have a pet feature added to DuckDB, it would be some form of native ordered table. In a database like Clickhouse or any of the dedicated time series DBMSes or log stores, there’s a built-in concept that a table might have an order, and the database will optimize based on the order. But, for databases that are logically just bags of rows (traditional DBMSes and also DuckDB [0]), you either need an index or you need to rely on full table scans or at least scans of big blocks. DuckDB does the latter really well, but I think it would be quite nice for some workflows to have explicit ordering. Also, I bet compression could work a lot better with ordering hints.

All that being said, I’m quite excited about DuckDB 2.0. I want to give the improved VARIANT support a try.

[0] Documentation on DuckDB’s native format is rather sparse AFAICT. But the DDL has nothing resembling an ordered table.

Re: A Preview of DuckDB v2.0

#27

Post by amluto »

jeffbee wrote: post #23

"We reimplemented ICU" U+1F631 FACE SCREAMING IN FEAR

I’m currently contemplating that MySQL apparently cannot do an INSTANT change of the collation of an unindexed column, even though, AFAICT, it has no effect whatsoever on the on-disk format or any data structure at all except for the metadata saying what the column type is.

I do not enjoy dealing with text encodings and collations in databases.

Re: A Preview of DuckDB v2.0

#28

Post by abirch »

fifilura wrote: post #20
jtbaker wrote: post #8

DuckDB is one of the things I've been most excited about in a long time. Introduced it to projects at 3 companies since 2023, greatly lowering resource requirements and running it in a variety of environments. Just having the ability to do out of core bigger than memory data processing on lower end consumer grade hardware is remarkable. Thanks to the team for everything!

My favourite is AWS Athena (backed by Trino). "If we use this we get indefinite RAM indefinite CPU and do not need to host a server". I had an impression that DuckDB was not great at distributing work to other machines, but good at doing it locally? Am I wrong?

DuckDB out of the box may not be great. But you have DuckLake, Quack, and even DeepSeek made their own distributed DB based on DuckDB: https://github.com/deepseek-ai/smallpond

Re: A Preview of DuckDB v2.0

#29

Post by fg137 »

throw1234567891 wrote: post #17
drums8787 wrote: post #13

Earlier quoted context omitted.

We use WASM DuckDB as the target for an in-browser agentic feature. Generated SQL runs against the user's individual tables that then feed in-browser dashboards. Excellent performance.

[flagged]

Running duckdb as wasm in browser for dashboards is a very common use case.

Does that make this account an alias as well?

Re: A Preview of DuckDB v2.0

#30

Post by aleda145 »

Excited about a stable C++ API for extensions!

I made a dry run extension a few months ago (https://github.com/aleda145/duckdb-dryrun), will be so nice to build it just once and know that it will always work.

Also urge anyone to make an extension, the template makes it quite smooth: https://github.com/duckdb/extension-template

Post reply on HN