Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB 0.7.0

duckdb.org

Post reply on HN
Print view View original

41–50 of 66 posts

Re: DuckDB 0.7.0

#41

Post by talolard »

talolard wrote: post #40

I use Duckdb as a data scientist / analyst. It’s amazing for working with large data locally, because it is very fast and there is almost 0 overhead for use. For example, I helped an Israeli ngo analyze retailer pricing data (supermarkets must publish prices every day by law). Pandas chokes on data that large, Postgres can handle it but aggregations are very slow. Duckdb is lightning fast. The traditional alternative…

Oh just took a look at the release notes, the new ability to write hive partitioned data with the partition by clause makes etl stuff much easier

Re: DuckDB 0.7.0

#42

Post by chrisjc »

santiagobasulto wrote: post #34
chrisjc wrote: post #25

Earlier quoted context omitted.

> in-process think of sqlite. > OLAP think data warehouse. Columnar for analytical workloads. If you want something "in-process" then you're probably going to have to decide between sqlite and duckdb. If your workload is 1) individual fast and frequent read-write operations (OLTP), then you should probably pick sqlite. 2) massive amounts of read-heavy analytical operations (OLAP), then you should probably pick duckdb…

How do you use DuckDB in production for a company? Store SQLite file in something like S3, sync it once per day and run DuckDB with it?

While DuckDB is an exciting and amazing project, I think the world that will open up around it is just as exciting, and these are exactly the kinds of questions that get me excited.

DuckDB is to Snowflake/BigQuery/DataBricks/etc...

what

sqlite is to MySQL/Postgres/Oracle/etc... (let's ignore for the moment that Postgres and Oracle have HTAP modes)

In other words, I don't think DuckDB aims to replace or compete against the big OLAP products/services such as Snowflake, BigQuery, DataBricks. Instead it's a natural and complementary component in the analytical stack.

Of course you'll see in the numerous blogs about how amazing it is for data exploration, wrangling, jupyter, pandas, etc... but personally I think the questions about how it could be used in production use-cases a lot more fascinating.

Data warehouses can become quite expensive to run and operate when you either have to allow

1) front-end analytical applications to connect to them directly to do analytics on the fly, or

2) if you pre-calculate ALL the analytics (whether they're used or not) that are offloaded to a cheaper and "faster" OLTP system.

I'm excited about how DuckDB can sort of bridge these two solutions.

1) Prepare semi-pre-calculated data on your traditional data warehouse. (store in internal table or external table like iceberg, delta, etc)

2) Ingest the subsets of this data needed for different production workloads in to DuckDB for last-mile analytics and slicing/dicing.

DuckDb could either interact with your

1) push-down queries to internal tables via their database scanners (arrow across the wire. postgres_scanner, hopefully more to come), or

2) prune external tables (iceberg, delta, etc) to get the subsets (interact with catalogs) of semi-pre-calculated analytical data on demand. Think intelligently partitioned parquet files on S3.

Last-mile analytics, pagination, etc can all be done within DuckDb either directly on your browser (WASM) or on the edge with something like AWS Lambda. This could and hopefully will result in reducing the cost of keeping data warehouses around to serve up fully pre-calculated analytics to consumers as well as reducing the complexity of your analytics stack/arch.

Re: DuckDB 0.7.0

#43

Post by chrisjc »

SOLAR_FIELDS wrote: post #22
shanipribadi wrote: post #15

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Online_analytical_processing as opposed to https://en.wikipedia.org/wiki/Online_transaction_processing DuckDB is when you need to do OLAP analysis, and the data fits in a single node (your laptop), but it's too large for plain excel. technically you can use PG/MySQL/Python+Numpy+Pandas to process those data for that use case as well, but DuckDB does it easier/faster most of the time.

What do frontends for this type of stuff look like these days? I remember one of my first jobs out of college they had wired up an OLAP cube into Excel so that you could import it into a pivot table and arbitrarily slice and dice data by dragging and dropping columns. I thought that was the coolest thing. Is there a modern day non excel equivalent of this? Cube.js?

I'd imagine Looker, PowerBI, Tableau are probably the front-ends you'd most commonly hear about, although I'm not sure DuckDb is supported or acknowledged by the parent companies (yet).

Re: DuckDB 0.7.0

#44

Post by polyrand »

I'm super excited about the new query building. It's like having CTEs that you can easily debug and explore, and instead of having to work with big queries now you can just play around with Python objects. This will make testing complex SQL easier too, you can do `.limit(20).show()` on any intermediate relation and look at the table.

Re: DuckDB 0.7.0

#45

Post by chrisjc »

merricksb wrote: post #33

Major discussion of project 2 days ago: https://news.ycombinator.com/item?id=34741195 (160 points/2 days ago/97 comments) Also: https://news.ycombinator.com/item?id=33612898 – DuckDB 0.6.0 (36 points/89 days ago) https://news.ycombinator.com/item?id=31355050 – Friendlier SQL with DuckDB (366 points/9 months ago/133 comments) And many others as per dang's comment: https://news.ycombinator.com/item?id=34746724

Any idea why this has been marked a dupe? I can't find the 0.7.0 story announced anywhere else on HN? The additions to 0.7.0 are quite significant and definitely news/discussion worthy.

I would have been upset missing this announcement and related commentary if I hadn't seen it before being marked a dupe.

Re: DuckDB 0.7.0

#47

Post by cmdlineluser »

chrisjc wrote: post #45
merricksb wrote: post #33

Major discussion of project 2 days ago: https://news.ycombinator.com/item?id=34741195 (160 points/2 days ago/97 comments) Also: https://news.ycombinator.com/item?id=33612898 – DuckDB 0.6.0 (36 points/89 days ago) https://news.ycombinator.com/item?id=31355050 – Friendlier SQL with DuckDB (366 points/9 months ago/133 comments) And many others as per dang's comment: https://news.ycombinator.com/item?id=34746724

Any idea why this has been marked a dupe? I can't find the 0.7.0 story announced anywhere else on HN? The additions to 0.7.0 are quite significant and definitely news/discussion worthy. I would have been upset missing this announcement and related commentary if I hadn't seen it before being marked a dupe.

Many of the previous duckdb threads have comments advertising Clickhouse features.

This was #1 before disappearing and now the current #1 is a blog post about Clickhouse.

Re: DuckDB 0.7.0

#49

Post by amalter »

chrisjc wrote: post #42
santiagobasulto wrote: post #34

Earlier quoted context omitted.

How do you use DuckDB in production for a company? Store SQLite file in something like S3, sync it once per day and run DuckDB with it?

While DuckDB is an exciting and amazing project, I think the world that will open up around it is just as exciting, and these are exactly the kinds of questions that get me excited. DuckDB is to Snowflake/BigQuery/DataBricks/etc... what sqlite is to MySQL/Postgres/Oracle/etc... (let's ignore for the moment that Postgres and Oracle have HTAP modes) In other words, I don't think DuckDB aims to replace or compete agains…

Do you work on my team? This is exactly how we're using Duckdb with Databricks as the massive data bulldozer and Duckdb as the scalpel.

Re: DuckDB 0.7.0

#50

Post by chrisjc »

amalter wrote: post #49
chrisjc wrote: post #42

Earlier quoted context omitted.

While DuckDB is an exciting and amazing project, I think the world that will open up around it is just as exciting, and these are exactly the kinds of questions that get me excited. DuckDB is to Snowflake/BigQuery/DataBricks/etc... what sqlite is to MySQL/Postgres/Oracle/etc... (let's ignore for the moment that Postgres and Oracle have HTAP modes) In other words, I don't think DuckDB aims to replace or compete agains…

Do you work on my team? This is exactly how we're using Duckdb with Databricks as the massive data bulldozer and Duckdb as the scalpel.

Definitely not since we use Snowflake, not Databricks. I'd love to hear more about your solution though!
Post reply on HN