Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Preview of DuckDB v2.0

duckdb.org

Post reply on HN
Print view View original

61–70 of 144 posts

Re: A Preview of DuckDB v2.0

#61

Post by jtbaker »

jayct wrote: post #57
arealaccount wrote: post #11

Earlier quoted context omitted.

We use DuckDB WASM with parquet to build dashboards in-browser. It's cool to be able to write SQL directly in a browser and not have to rely on REST/Graphql/etc to access the data layer.

curious if you're using something mostly-out-of-the-box to layer on visualizations for your dashboards? relatively new to duckdb, love it so far, looking at alternatives for downstream visualization. so far just exporting datasets and piping into python scripts.

For a schema-first (vs. code first) approach (which I think would be a sweet spot for agent driven dashboarding), I'd suggest looking at https://vega.github.io/vega-lite/ or https://vega.github.io/vega/. A little higher level than full D3 but gives you a little higher level approach.

Re: A Preview of DuckDB v2.0

#63

Post by allan_s »

mediaman wrote: post #36
otter-in-a-suit wrote: post #32

Super excited about Quack (partially due to the name). I use duckdb for both analytics and runtime, but I do have to serve/handle/manage a giant, multi-GiB duckdb file as effectively a runtime artifact[1]. I'm aware that this isn't the _perfect_ database for this, but the mix of it being fast, having spatial support, sane coding interfaces, great dbt integration, and me being able to do everything between "run a gian…

I built a platform for some midsize companies in a specific vertical that is basically a data warehouse with some LLM-driven dashboarding and query tools on top. Typical data size 5-150gb. So I built a service layer around duckdb, where each tenant gets their own duckdb. I'm also in the boat of knowing that duckdb is not the perfect solution for this (the classic use case is running it against local data on a laptop)…

As somebody new to this and with a use case very similar to yours , what would have been a more suitable solution for this ?

The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?

Re: A Preview of DuckDB v2.0

#65

Post by staticautomatic »

jtbaker wrote: post #25
throwaw12 wrote: post #9

Earlier quoted context omitted.

Curious to learn more about how people are using it? Are they downloading parquet files and running analyses locally, or are they connecting to Iceberg-like data lake and leveraging DuckDBs query engine capabilities or have you exposed an interface (REST, UI) to query your data?

I've got a couple of different use cases: - ETL pipelines running on K8s nodes. Using their streaming processing engine means I can run smaller pods/nodes if needed, for datasets that may have required large dataframe-like transformations that may have buffered a big dataset into memory previously. - A CLI distributed to an internal team to do a postprocessing step on a large modeling dataset - to get it into a consu…

Similar here. Lots of places where we replaced Pandas with DuckDB for transformations. Also have scriptable custom dashboards running on top of BigQuery data pre-aggregated and extracted to parquet on GCS. It's way faster and the only limiting factor is your viz library. It was pretty easy to build and the only big gotcha I encountered was finding, somewhat counter-intuitively, that it's often best minimize partitioning.

Re: A Preview of DuckDB v2.0

#66

Post by te_chris »

cheema33 wrote: post #64

With some of these changes, it appears to be encroaching on clickhouse territory. Or are they still very different products?

Well, I can tell you this is making me actively reconsider dropping duckdb for ch as we go from prototype to prod

Re: A Preview of DuckDB v2.0

#67

Post by jeremyjh »

Tomte wrote: post #12
ethagnawl wrote: post #10

Earlier quoted context omitted.

UPDATE: Removed uninformed comment about the relationship between DuckDB and MotherDuck.

Wait, what? I‘ve always assumed MotherDuck is the company by the main developers of DuckDB? They are unrelated?

They launched ducklake: https://ducklake.select/

Re: A Preview of DuckDB v2.0

#68

Post by encoderer »

cheema33 wrote: post #64

With some of these changes, it appears to be encroaching on clickhouse territory. Or are they still very different products?

Definitely encroaching.

Our last product (SaaS observability) uses Clickhouse.

Our next product (self-hosted observability) uses DuckDB.

Re: A Preview of DuckDB v2.0

#69

Post by ChillyCapy »

Great work!

I built a browser tool for querying local Parquet, CSV, JSON, Excel, Arrow, Avro, DBF, and SQLite files with DuckDB-WASM.

Most probably after DuckDB v2.0 release I will revamp my tool as well.

Re: A Preview of DuckDB v2.0

#70

Post by rrgok »

Sometimes I prefer DuckDB query language to MySQL or Postgres. It is not even about performance, it just feels right. I just connect to my DBS from DuckDB just to use its query language. And besides it gives you a single language frontend.
Post reply on HN