Hate to bring it up, but 10,000 commits in less than 6 months is a lot. Is AI a major contribute here?
Is AI use for accelerated development of a beloved tool like DuckDB enough to quiet lingering doubters?
A Preview of DuckDB v2.0
81–90 of 144 posts
Re: A Preview of DuckDB v2.0
#82Earlier quoted context omitted.
I built a platform for some midsize companies in a specific vertical that is basically a data warehouse with some LLM-driven dashboarding and query tools on top. Typical data size 5-150gb. So I built a service layer around duckdb, where each tenant gets their own duckdb. I'm also in the boat of knowing that duckdb is not the perfect solution for this (the classic use case is running it against local data on a laptop)…
As somebody new to this and with a use case very similar to yours , what would have been a more suitable solution for this ? The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?
Clickhouse has a more intentionally built ingestion system. Duckdb has concurrency limits so you can't have a writer and a reader on the same file if they're not the same process (multiple readers is fine).
But that's not too hard to work around. You can either have a single process that owns both writing and reading that file, or you can do a data lake where you post updates as parquet files into object storage, and duckdb handles the catalog. The Quack protocol also basically fixes this (though still in beta).
With Clickhouse, you can of course still have tenant separation, but you have to do it by managing users within Clickhouse that map to users/tenants of your main app, so that you can restrict SQL access by tenant to only their data store. Not a huge deal but I just like the Unix "it's just a file" simplicity of "Tenant A gets to run arbitrary SQL against their separate read-only, no-ATTACH duckdb file".
Re: A Preview of DuckDB v2.0
#83Earlier quoted context omitted.
As somebody new to this and with a use case very similar to yours , what would have been a more suitable solution for this ? The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?
Clickhouse has a more intentionally built ingestion system. Duckdb has concurrency limits so you can't have a writer and a reader on the same file if they're not the same process (multiple readers is fine). But that's not too hard to work around. You can either have a single process that owns both writing and reading that file, or you can do a data lake where you post updates as parquet files into object storage, and…
DuckDB's sweet spot is for 'smallish' lakehouses. So, ingestion should not be via duckdb, but rather something like Python/DLTHub for small scale or Spark for large scale or Kafka/Debezium/Flink for streaming data.
Re: A Preview of DuckDB v2.0
#84Earlier quoted context omitted.
Check it out Arc, I think that we can help there, plus, use DuckDB as a engine: https://github.com/Basekick-Labs/arc
Hey this is neat and I just found out you're from BA, Argentina. I would love you be part of dataengine.ar and maybe talk about arc?
Hey hey. I love to. I'm from Argentina, and I say that Arc is proudly make from Latam, but I live in Costa Rica.
Re: A Preview of DuckDB v2.0
#85I <3 DuckDB. It has become one of my go to tools for storing, data processing , integrations and now even graph. More importantly it's fun to use because it is so portable. Looking forward to v2.
What advantages does it have over SQLite in your use cases? Can you give any examples?
Re: A Preview of DuckDB v2.0
#86DuckDB is one of the things I've been most excited about in a long time. Introduced it to projects at 3 companies since 2023, greatly lowering resource requirements and running it in a variety of environments. Just having the ability to do out of core bigger than memory data processing on lower end consumer grade hardware is remarkable. Thanks to the team for everything!
Curious to learn more about how people are using it? Are they downloading parquet files and running analyses locally, or are they connecting to Iceberg-like data lake and leveraging DuckDBs query engine capabilities or have you exposed an interface (REST, UI) to query your data?
I'm using duckdb/duckdb-go as query engine for my Go services: moving hot data from Postgres to Parquet files on S3 or to Iceberg; querying cold data on Iceberg, ... instead of using different Go libraries.
Re: A Preview of DuckDB v2.0
#87DuckDB is one of the things I've been most excited about in a long time. Introduced it to projects at 3 companies since 2023, greatly lowering resource requirements and running it in a variety of environments. Just having the ability to do out of core bigger than memory data processing on lower end consumer grade hardware is remarkable. Thanks to the team for everything!
Curious to learn more about how people are using it? Are they downloading parquet files and running analyses locally, or are they connecting to Iceberg-like data lake and leveraging DuckDBs query engine capabilities or have you exposed an interface (REST, UI) to query your data?
I use it locally with parquet files
Re: A Preview of DuckDB v2.0
#88> A repository is a name, a URL prefix, and one or more RSA public keys that are trusted to sign the extensions served from it.
Is it too late to beg Hannes and Mark to let us have something like minisign instead of RSA? :)
Very excited for 2.0, congrats folks!
Re: A Preview of DuckDB v2.0
#89It's funny to me that we still don't have incremental materialized views. All of the parts are there (export state, agg_state (forget fn name), finalize). I wonder if they're avoiding an explicit war with clickhouse or something. I do recall they mentioned they want to add this to ducklake.
Incremental MVs are ClickHouse's best feature. If DDB adds this, the last moat is distributed query execution.
Re: A Preview of DuckDB v2.0
#90Earlier quoted context omitted.
As somebody new to this and with a use case very similar to yours , what would have been a more suitable solution for this ? The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?
Clickhouse has a more intentionally built ingestion system. Duckdb has concurrency limits so you can't have a writer and a reader on the same file if they're not the same process (multiple readers is fine). But that's not too hard to work around. You can either have a single process that owns both writing and reading that file, or you can do a data lake where you post updates as parquet files into object storage, and…
Ducklake supports postgres for the catalog, so you get the postgres concurrency benefits + duckdb engine to read the parquet files in the bucket.