101–110 of 144 posts
Super excited about Quack (partially due to the name). I use duckdb for both analytics and runtime, but I do have to serve/handle/manage a giant, multi-GiB duckdb file as effectively a runtime artifact[1]. I'm aware that this isn't the _perfect_ database for this, but the mix of it being fast, having spatial support, sane coding interfaces, great dbt integration, and me being able to do everything between "run a gian…
I am really looking forward to this hitting v2.0. I can't stand uncompressed JSON - so space-inefficient. But heterogenous JSON in parquet files is such a pain because of schema differences causing fields to be silently dropped. Having DuckDB solve this is exactly what I've been looking for.
Super excited about Quack (partially due to the name). I use duckdb for both analytics and runtime, but I do have to serve/handle/manage a giant, multi-GiB duckdb file as effectively a runtime artifact[1]. I'm aware that this isn't the _perfect_ database for this, but the mix of it being fast, having spatial support, sane coding interfaces, great dbt integration, and me being able to do everything between "run a gian…
Earlier quoted context omitted.
You might know this already, but you can query pandas/polars/arrow tables directly w/ duckdb and use whatever stats packages you feel like alongside it in the same python script. I feel like they do a decent job sticking to the simpler statistical fans that make sense in sql.
Sorry, I meant duckdb-cli, not duckdb in Python.
https://duckdb.org/community_extensions/extensions/stats_duc...
https://duckdb.org/community_extensions/extensions/stochasti...
Would be really cool if they were to add statistical functions too. I'd jump at the chance of getting to use this over pandas
Earlier quoted context omitted.
As somebody new to this and with a use case very similar to yours , what would have been a more suitable solution for this ? The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?
another variant: i put duckdb on a lambda and pointed it at s3 for the data. my data was closer to 2GB but the queries were quick and nearly free with superset pointed at it is your setup running into problems that makes you need something more?
Earlier quoted context omitted.
Sorry, I meant duckdb-cli, not duckdb in Python.
Oh, then you might like these DuckDB community extensions that provide stats functions! https://duckdb.org/community_extensions/extensions/stats_duc... https://duckdb.org/community_extensions/extensions/stochasti...
Would be really cool if they were to add statistical functions too. I'd jump at the chance of getting to use this over pandas
which statistical functions do you want? i’m curious because i love duckdb and use it for a variety of projects but always want to learn more about how to use tools better.
Earlier quoted context omitted.
Clickhouse has a more intentionally built ingestion system. Duckdb has concurrency limits so you can't have a writer and a reader on the same file if they're not the same process (multiple readers is fine). But that's not too hard to work around. You can either have a single process that owns both writing and reading that file, or you can do a data lake where you post updates as parquet files into object storage, and…
DuckDB's sweet spot is for 'smallish' lakehouses. So, ingestion should not be via duckdb, but rather something like Python/DLTHub for small scale or Spark for large scale or Kafka/Debezium/Flink for streaming data.
For our community DuckDB is the default data warehouse for local development environment. Last month +90,000 users used dlt (and their AI code editor) to load data into DuckDB.
Because of our proximity to the DuckDB community we are seeing enterprise DuckDB usage first hand. People imo sleep on the data volumes DuckDB can handle. We see Fortune 100 companies use dlt and DuckDB in production on their Lakehouses in hybrid cloud deployments. I can eg mention Stellantis (Chrysler, Jeep, Peugeot etc) because they talk about it publicly.