Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Preview of DuckDB v2.0

duckdb.org

Post reply on HN
Print view View original

91–100 of 144 posts

Re: A Preview of DuckDB v2.0

#91

Post by briHass »

d3Xt3r wrote: post #85
srameshc wrote: post #5

I <3 DuckDB. It has become one of my go to tools for storing, data processing , integrations and now even graph. More importantly it's fun to use because it is so portable. Looking forward to v2.

What advantages does it have over SQLite in your use cases? Can you give any examples?

Not OP, but for me, the lack of essentially any type system in SQLite makes it a total no-go for storing data long-term or that more than one application needs to access. Date/time being an especially painful footgun in SQLite.

I view SQLite as something a single application can use for storing state/settings/misc operational data instead of directly writing files, especially if the data being stored is relational or needs ACID. As soon as the data itself has meaning and structure per se, you're better off with something that can help enforce and describe the data: rich datatypes, foreign keys that aren't optional, etc.

Re: A Preview of DuckDB v2.0

#92

Post by ghshephard »

dangoodmanUT wrote: post #89

It's funny to me that we still don't have incremental materialized views. All of the parts are there (export state, agg_state (forget fn name), finalize). I wonder if they're avoiding an explicit war with clickhouse or something. I do recall they mentioned they want to add this to ducklake. Incremental MVs are ClickHouse's best feature. If DDB adds this, the last moat is distributed query execution.

Have you run into scenarios where a simple view doesn't accomplish what you require? I always feel like views do everything I want - because the speed is so great, the full recompute isn't that big a deal.

Maybe it's a bigger deal when you have multiple users/and or more repeated queries against something that's really expensive?

Re: A Preview of DuckDB v2.0

#93

Post by 8note »

allan_s wrote: post #63
mediaman wrote: post #36

Earlier quoted context omitted.

I built a platform for some midsize companies in a specific vertical that is basically a data warehouse with some LLM-driven dashboarding and query tools on top. Typical data size 5-150gb. So I built a service layer around duckdb, where each tenant gets their own duckdb. I'm also in the boat of knowing that duckdb is not the perfect solution for this (the classic use case is running it against local data on a laptop)…

As somebody new to this and with a use case very similar to yours , what would have been a more suitable solution for this ? The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?

another variant:

i put duckdb on a lambda and pointed it at s3 for the data. my data was closer to 2GB but the queries were quick and nearly free with superset pointed at it

is your setup running into problems that makes you need something more?

Re: A Preview of DuckDB v2.0

#95

Post by nh2 »

therealdrag0 wrote: post #81

Hate to bring it up, but 10,000 commits in less than 6 months is a lot. Is AI a major contribute here? Is AI use for accelerated development of a beloved tool like DuckDB enough to quiet lingering doubters?

If you merge PRs that have commit mesages like this, it's easy to arrive at 10000 commits in 6 months:

    rename to NodePointer instead
    format
    Revert "format"
    Revert "rename to NodePointer instead"
    rename to OptionalNodePtr
    woops
    update comment
    slot renames
    more renames
Source: https://github.com/duckdb/duckdb/pull/23605

If every Ctrl+S is a commit, it'll go up fast.

"woops"!

Re: A Preview of DuckDB v2.0

#96

Post by Pedro_Ribeiro »

ghshephard wrote: post #92
dangoodmanUT wrote: post #89

It's funny to me that we still don't have incremental materialized views. All of the parts are there (export state, agg_state (forget fn name), finalize). I wonder if they're avoiding an explicit war with clickhouse or something. I do recall they mentioned they want to add this to ducklake. Incremental MVs are ClickHouse's best feature. If DDB adds this, the last moat is distributed query execution.

Have you run into scenarios where a simple view doesn't accomplish what you require? I always feel like views do everything I want - because the speed is so great, the full recompute isn't that big a deal. Maybe it's a bigger deal when you have multiple users/and or more repeated queries against something that's really expensive?

I've had some use cases with low latency requirements and frequent data updates. If you want to serve data on a frontend fast it's often not feasible to recompute everything every time

Re: A Preview of DuckDB v2.0

#97

Post by otter-in-a-suit »

allan_s wrote: post #63
mediaman wrote: post #36

Earlier quoted context omitted.

I built a platform for some midsize companies in a specific vertical that is basically a data warehouse with some LLM-driven dashboarding and query tools on top. Typical data size 5-150gb. So I built a service layer around duckdb, where each tenant gets their own duckdb. I'm also in the boat of knowing that duckdb is not the perfect solution for this (the classic use case is running it against local data on a laptop)…

As somebody new to this and with a use case very similar to yours , what would have been a more suitable solution for this ? The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?

There's a few options.

Clickhouse, as I mentioned, can be a good final layer, as can postgres.

You can still use duckdb for intermediate transformations, even if the final data lives elsewhere.

duckdb can also access various external sources, such as s3, so you could use duckdb for transformations and write "classic" parquet files to S3 and query them with an engine of your choice (which, again, could also be duckdb, but nothing stopping you from using Trino or something along those lines).

All a question of scale, complexity, cost, and latency. For reasonably low latency, shipping a duckdb file to the edge is fine, I think. Makes CI/deployments more complicated. Or you could assemble the actual duckdb file on site - probably easier with K8s and an init container that can scale? Something like that, I don't use K8s for SkaldMaps, but I have experimented a bit.

For SkaldMaps, the backend is written in go and has an abstraction to plug in a different presentation data store, so I would just need to re-wire data platform to write the final tables to e.g. CH instead of duckdb.

Re: A Preview of DuckDB v2.0

#98

Post by andyferris »

I was curious to see they are advertising OLTP-like transactional processing speed. It would be super convenient to have one DB for OLTP and OLAP purposes!

Has anybody here tried using it that way?

(though I don't see any way to deal with write skew and other transactional guarantees - no SERIALIZABLE optimisitic concurrency, no SELECT FOR UPDATE pessimistic concurrency, etc)

Re: A Preview of DuckDB v2.0

#99

Post by dartharva »

biophysboy wrote: post #79
dartharva wrote: post #54

Would be really cool if they were to add statistical functions too. I'd jump at the chance of getting to use this over pandas

You might know this already, but you can query pandas/polars/arrow tables directly w/ duckdb and use whatever stats packages you feel like alongside it in the same python script. I feel like they do a decent job sticking to the simpler statistical fans that make sense in sql.

Sorry, I meant duckdb-cli, not duckdb in Python.
Post reply on HN