Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB 0.7.0

duckdb.org

Post reply on HN
Print view View original

51–60 of 66 posts

Re: DuckDB 0.7.0

#51

Post by dang »

chrisjc wrote: post #45
merricksb wrote: post #33

Major discussion of project 2 days ago: https://news.ycombinator.com/item?id=34741195 (160 points/2 days ago/97 comments) Also: https://news.ycombinator.com/item?id=33612898 – DuckDB 0.6.0 (36 points/89 days ago) https://news.ycombinator.com/item?id=31355050 – Friendlier SQL with DuckDB (366 points/9 months ago/133 comments) And many others as per dang's comment: https://news.ycombinator.com/item?id=34746724

Any idea why this has been marked a dupe? I can't find the 0.7.0 story announced anywhere else on HN? The additions to 0.7.0 are quite significant and definitely news/discussion worthy. I would have been upset missing this announcement and related commentary if I hadn't seen it before being marked a dupe.

A moderator downweighted it because there was a DuckDB thread on the front page for 16 hours just a couple days ago:

DuckDB – An in-process SQL OLAP database management system - https://news.ycombinator.com/item?id=34741195 - Feb 2023 (99 comments)

HN operates on the basis of not having too much repetition on the front page. As seen at https://news.ycombinator.com/item?id=34746724, there have also been lots of other DuckDB threads in recent months.

I realize a new release is rightly significant to the people working on the product and/or who are users of the product, and it would have been better for the major thread not to just be a generic post about the project. However, that distinction isn't as salient from a HN discussion point of view, because either way, the thread will fill up with comments about the product in general. You can see that quite clearly in the current thread. The important criterion from an HN point of view is "is this submission different enough to support a substantially different discussion", and in this case the answer is no, so the moderation call was correct.

It's quite impossible to learn about every major release of every major product from HN—frontpage space is the scarcest resource we have [1]. The front page could consist of nothing else and you still couldn't learn about them all from HN alone. Nor is that the purpose of the site; the purpose is intellectual curiosity [2]. Curiosity doesn't do well with repetition [2], so the median curious reader isn't served by having two big threads about the same product within days. Of course, we all have at least one project where we would love to see that, but it's a different choice in everyone's case and we have to try to serve everybody.

[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

[2] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...

[3] https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...

Re: DuckDB 0.7.0

#52

Post by stinos »

talolard wrote: post #40

I use Duckdb as a data scientist / analyst. It’s amazing for working with large data locally, because it is very fast and there is almost 0 overhead for use. For example, I helped an Israeli ngo analyze retailer pricing data (supermarkets must publish prices every day by law). Pandas chokes on data that large, Postgres can handle it but aggregations are very slow. Duckdb is lightning fast. The traditional alternative…

So I'm not super familiar with different databases, but do understand the basics and do know how to work wit data with e.g. pandas, and do think I understand what Duckdb is useful for, but what I'm still completely missing is: how do I get data in Duckdb? I.e. how did you get that data into Duckdb? Or: suppose I have a device producing sensor data, normally I'd connect to some MySQL endpoint somehow and tell it to insert data. How does one do that with Duckdb? Or is the idea rather that you construct your Duckdb first by getting data from somewhere else (like the MySQL db in my example)?

Re: DuckDB 0.7.0

#53

Post by chrisjc »

dang wrote: post #51
chrisjc wrote: post #45

Earlier quoted context omitted.

Any idea why this has been marked a dupe? I can't find the 0.7.0 story announced anywhere else on HN? The additions to 0.7.0 are quite significant and definitely news/discussion worthy. I would have been upset missing this announcement and related commentary if I hadn't seen it before being marked a dupe.

A moderator downweighted it because there was a DuckDB thread on the front page for 16 hours just a couple days ago: DuckDB – An in-process SQL OLAP database management system - https://news.ycombinator.com/item?id=34741195 - Feb 2023 (99 comments) HN operates on the basis of not having too much repetition on the front page. As seen at https://news.ycombinator.com/item?id=34746724 , there have also been lots of other…

Totally understand and appreciate the reasons why!

Re: DuckDB 0.7.0

#54

Post by smt88 »

nurettin wrote: post #21
smt88 wrote: post #5

This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?

Might be wrong, but it looks like duckdb lets you host the database engine in your process, so you don't pay for IPC. It is the opposite of Redis, as you use it to share memory between processes

I'm confused by your sentence structure. Are you saying DuckDB lets you share memory between processes?

If so, that's the opposite of what DuckDB does. Under to "When not to use DuckDB" section of their website, they say:

> "[Do not use DuckDB when] writing to a single database from multiple concurrent processes"

Honestly, that's the most baffling part. I can't imagine wanting any database that's locked in a single process.

Re: DuckDB 0.7.0

#55

Post by smt88 »

stagg wrote: post #24
smt88 wrote: post #5

This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?

We're using it to migrate data pipelines in AWS which were previously run using Glue to Lambda with duckdb. Glue was too heavyweight, slow and expensive for our GB data volumes. We consume csv files use a lambda and duckdb to convert them to parquet. Then another lambda to load these parquet files and do our transformation logic (deduplications, enrichments, clean up, etc) and writing out to parquet files.

Hmm, interesting... so basically DuckDB works in this case because there's no way to parallelize the migration of a single volume anyway?

This is definitely a pretty niche case, though, so there must be something more general that this was built to do.

Re: DuckDB 0.7.0

#56

Post by smt88 »

amelius wrote: post #14
smt88 wrote: post #5

This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?

From what I can gather, it's in-process, so less of a hassle when installing things or doing system administration. Also I suppose easier to port, and e.g. run in a browser. I guess any speed improvements wrt a DB running in a separate process will be tiny and irrelevant for most applications.

> easier to port, and e.g. run in a browser

This has to be the main point, right? DuckDB isn't the first mover here (SQL.js, which is SQLite compiled to WASM using emscripten, seems to work fine), but perhaps DuckDB is better as a purpose-built solution.

Re: DuckDB 0.7.0

#57

Post by smt88 »

shanipribadi wrote: post #18
adlpz wrote: post #12

Earlier quoted context omitted.

I'm interested in this, too. I can totally see how not having to manage a standalone RDBMS makes sense. But, what's the real-world advantage over something like SQLite? I mean, the idea of an in-memory relational engine for things like games or embedded totally makes sense, but this seems to target large datasets and deep analysis. As far as I understand with this model you pretty much re-ingest data from the "raw" s…

think BI tools, analytics dashboards for exploratory analysis, or even just exploratory analysis on the terminal with it's rich query capabilities. you can keep analytics data in SQLite, but DuckDB will process it faster/easier for the analytics use cases.

> think BI tools, analytics dashboards for exploratory analysis, or even just exploratory analysis on the terminal with it's rich query capabilities

I thought about that, but I'd never use DuckDB for it because DuckDB is locked into a single process. I can't figure out a benefit of being suck with one core when I always have between 2 and 32 available to me.

Re: DuckDB 0.7.0

#58

Post by smt88 »

adlpz wrote: post #12
smt88 wrote: post #5

This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?

I'm interested in this, too. I can totally see how not having to manage a standalone RDBMS makes sense. But, what's the real-world advantage over something like SQLite? I mean, the idea of an in-memory relational engine for things like games or embedded totally makes sense, but this seems to target large datasets and deep analysis. As far as I understand with this model you pretty much re-ingest data from the "raw" s…

> But, what's the real-world advantage over something like SQLite?

This very specific question is what I'm trying to understand. SQLite can be run in memory and as a temporary store.

Re: DuckDB 0.7.0

#59

Post by smt88 »

isoprophlex wrote: post #13
smt88 wrote: post #5

This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?

Quick, in-core data transformation. If you want to transform some data right now, one option is writing pyspark and running that on a spark cluster. But noone really has big big data, there are relatively few cases where you have multi TB datasets, warranting the complexities of running the analyics in a distributed way. DuckDB lets you process all that locally. It's the OLAP equivalent to SQLite's OLTP. If I wasn't…

Interesting. I need to think about this one a little bit. Thank you.

Is there a reason "in-core" is a specific requirement here?

Re: DuckDB 0.7.0

#60

Post by tylerhannan »

santiagobasulto wrote: post #34
chrisjc wrote: post #25

Earlier quoted context omitted.

> in-process think of sqlite. > OLAP think data warehouse. Columnar for analytical workloads. If you want something "in-process" then you're probably going to have to decide between sqlite and duckdb. If your workload is 1) individual fast and frequent read-write operations (OLTP), then you should probably pick sqlite. 2) massive amounts of read-heavy analytical operations (OLAP), then you should probably pick duckdb…

How do you use DuckDB in production for a company? Store SQLite file in something like S3, sync it once per day and run DuckDB with it?

It's an interesting question... DuckDB is a library. Just like SQLite is a library.

It's not designed for concurrent queries, clients connecting to a database, etc. I know there will be companies built around that problem space.

If "serverless" database is a thing, is there a category of software that is "production-less"? :)

(The above is a joke, lol.)

Post reply on HN