Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Preview of DuckDB v2.0

duckdb.org

Post reply on HN
Print view View original

41–50 of 144 posts

Re: A Preview of DuckDB v2.0

#41

Post by tccole »

throwaw12 wrote: post #9
jtbaker wrote: post #8

DuckDB is one of the things I've been most excited about in a long time. Introduced it to projects at 3 companies since 2023, greatly lowering resource requirements and running it in a variety of environments. Just having the ability to do out of core bigger than memory data processing on lower end consumer grade hardware is remarkable. Thanks to the team for everything!

Curious to learn more about how people are using it? Are they downloading parquet files and running analyses locally, or are they connecting to Iceberg-like data lake and leveraging DuckDBs query engine capabilities or have you exposed an interface (REST, UI) to query your data?

Yes.

I have used it with WASM for some web applications for web use. I have also used with locally for querying 100 gigs of data. And I have used it in the cloud as the serverless gold layer for Apache superset.

Re: A Preview of DuckDB v2.0

#42

Post by tccole »

jtbaker wrote: post #25
throwaw12 wrote: post #9

Earlier quoted context omitted.

Curious to learn more about how people are using it? Are they downloading parquet files and running analyses locally, or are they connecting to Iceberg-like data lake and leveraging DuckDBs query engine capabilities or have you exposed an interface (REST, UI) to query your data?

I've got a couple of different use cases: - ETL pipelines running on K8s nodes. Using their streaming processing engine means I can run smaller pods/nodes if needed, for datasets that may have required large dataframe-like transformations that may have buffered a big dataset into memory previously. - A CLI distributed to an internal team to do a postprocessing step on a large modeling dataset - to get it into a consu…

Hell yeah; a fellow sveltekit fan.

Re: A Preview of DuckDB v2.0

#43

Post by ostwilkens »

drannex wrote: post #35
badatnames wrote: post #22

Looks like an awesome release, but the smell of AI from that post is horrid. Here is a wild idea: is it really so hard to edit out sentences structured and punctuated like this - it's so painfully obvious and distracts from the content. The effect is real.

I don't really get an AI smell on this, in fact I see multiple parts that an AI would have corrected - grammatical issues, personal writing quirks, etc. Writing similar to this: is quite a normal way of writing for technical articles - especially when you are trying to make the point clear and well organized.

"A major version bump is not something we do lightly, and it is not just ceremony"

For me, this sentence pops out like a sore thumb

Re: A Preview of DuckDB v2.0

#44

Post by thejosh »

I've been working on a demo database project, and have been really impressed by the UI. So glad they decided to put more effort into it, it has made building a "follow along" tutorial really nice.

Re: A Preview of DuckDB v2.0

#46

Post by ignaciovdk »

otter-in-a-suit wrote: post #32

Super excited about Quack (partially due to the name). I use duckdb for both analytics and runtime, but I do have to serve/handle/manage a giant, multi-GiB duckdb file as effectively a runtime artifact[1]. I'm aware that this isn't the _perfect_ database for this, but the mix of it being fast, having spatial support, sane coding interfaces, great dbt integration, and me being able to do everything between "run a gian…

Check it out Arc, I think that we can help there, plus, use DuckDB as a engine: https://github.com/Basekick-Labs/arc

Re: A Preview of DuckDB v2.0

#47

Post by d33 »

Are there improvements in how memory_limit works? I often had DuckDB get OOM killed because it went beyond its limit. It's definitely one of the reasons why I usually have an AI tune the environment for my datasets.

Re: A Preview of DuckDB v2.0

#48

Post by dmix »

otter-in-a-suit wrote: post #32

Super excited about Quack (partially due to the name). I use duckdb for both analytics and runtime, but I do have to serve/handle/manage a giant, multi-GiB duckdb file as effectively a runtime artifact[1]. I'm aware that this isn't the _perfect_ database for this, but the mix of it being fast, having spatial support, sane coding interfaces, great dbt integration, and me being able to do everything between "run a gian…

> to "read/query a csv on disk"

I discovered DuckDB looking for a way to analyze Nginx access.log's and it's an amazing tool. I believe it should be a standard tool like ripgrep for devs.

Re: A Preview of DuckDB v2.0

#49

Post by ignaciovdk »

amluto wrote: post #26

If I could have a pet feature added to DuckDB, it would be some form of native ordered table. In a database like Clickhouse or any of the dedicated time series DBMSes or log stores, there’s a built-in concept that a table might have an order, and the database will optimize based on the order. But, for databases that are logically just bags of rows (traditional DBMSes and also DuckDB [0]), you either need an index or…

Arc does pruning, and make that scan faster. Check it out: https://github.com/Basekick-Labs/arc
Post reply on HN