Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB 0.7.0

duckdb.org

Post reply on HN
Print view View original

11–20 of 66 posts

Re: DuckDB 0.7.0

#11

Post by dirkderkdurk »

eliasmacpherson wrote: post #2

So say I wanted to try out a workload on various sql databases, mariadb, sqllite, postgres - is there a database that will act as a front end to them? I find this 'support for pluggable database engines' intriguing. Not least because I can then claim to have used all of the database engines in anger :-) (I know that it is probably a dumb question due to the following, but I asked anyway: https://en.wikipedia.org/wiki…

You could also try using ODBC or ADO.NET. I've not used the latter, but ODBC was my goto for this kind of thing a decade or so ago. So mileage may vary drastically, and there might be roadkill along the way.

https://en.wikipedia.org/wiki/Open_Database_Connectivity

Re: DuckDB 0.7.0

#12

Post by adlpz »

smt88 wrote: post #5

This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?

I'm interested in this, too.

I can totally see how not having to manage a standalone RDBMS makes sense. But, what's the real-world advantage over something like SQLite?

I mean, the idea of an in-memory relational engine for things like games or embedded totally makes sense, but this seems to target large datasets and deep analysis.

As far as I understand with this model you pretty much re-ingest data from the "raw" source on startup every time. Is this correct?

Judging by the rise on interest I'm sure there's an obvious use case I'm not seeing either.

Re: DuckDB 0.7.0

#13

Post by isoprophlex »

smt88 wrote: post #5

This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?

Quick, in-core data transformation. If you want to transform some data right now, one option is writing pyspark and running that on a spark cluster. But noone really has big big data, there are relatively few cases where you have multi TB datasets, warranting the complexities of running the analyics in a distributed way.

DuckDB lets you process all that locally. It's the OLAP equivalent to SQLite's OLTP.

If I wasn't so beholden to the vagaries and inefficiencies of C-level endorsed enterprise software, I'd immediately be trying this out for data transformations/pipelines. I think that one big box (200+ gb ram, couple of cores and fat IO/network) runs circles around an entire spark cluster.

Re: DuckDB 0.7.0

#14

Post by amelius »

smt88 wrote: post #5

This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?

From what I can gather, it's in-process, so less of a hassle when installing things or doing system administration. Also I suppose easier to port, and e.g. run in a browser. I guess any speed improvements wrt a DB running in a separate process will be tiny and irrelevant for most applications.

Re: DuckDB 0.7.0

#15

Post by shanipribadi »

0x008 wrote: post #9

> DuckDB is an in-process SQL OLAP database management system I don’t understand what it means. Can someone explain? I don’t get why they put such a complicated claim with unexplained acronyms on their homepage. When I shop for a db, when should I consider duck DB compared to for example Postgres or MySQL? Or do they compete with arrow or parquet? To me it’s unclear because they don’t say what they compete against.

https://en.wikipedia.org/wiki/Online_analytical_processing as opposed to https://en.wikipedia.org/wiki/Online_transaction_processing

DuckDB is when you need to do OLAP analysis, and the data fits in a single node (your laptop), but it's too large for plain excel.

technically you can use PG/MySQL/Python+Numpy+Pandas to process those data for that use case as well, but DuckDB does it easier/faster most of the time.

Re: DuckDB 0.7.0

#16

Post by colesantiago »

ddorian43 wrote: post #10
0x008 wrote: post #9

> DuckDB is an in-process SQL OLAP database management system I don’t understand what it means. Can someone explain? I don’t get why they put such a complicated claim with unexplained acronyms on their homepage. When I shop for a db, when should I consider duck DB compared to for example Postgres or MySQL? Or do they compete with arrow or parquet? To me it’s unclear because they don’t say what they compete against.

> I don’t understand what it means. It's like Sqlite(OLTP) but for OLAP.

This is still confusing, what do I use this for exactly?

Re: DuckDB 0.7.0

#17

Post by Jarwain »

0x008 wrote: post #9

> DuckDB is an in-process SQL OLAP database management system I don’t understand what it means. Can someone explain? I don’t get why they put such a complicated claim with unexplained acronyms on their homepage. When I shop for a db, when should I consider duck DB compared to for example Postgres or MySQL? Or do they compete with arrow or parquet? To me it’s unclear because they don’t say what they compete against.

I wouldn't say compete. DuckDB feels like it fits right between postgres and parquet. I think of it as like SQLite for columnar data. Sql semantics but optimized for operations against columns instead of rows.

Re: DuckDB 0.7.0

#18

Post by shanipribadi »

adlpz wrote: post #12
smt88 wrote: post #5

This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?

I'm interested in this, too. I can totally see how not having to manage a standalone RDBMS makes sense. But, what's the real-world advantage over something like SQLite? I mean, the idea of an in-memory relational engine for things like games or embedded totally makes sense, but this seems to target large datasets and deep analysis. As far as I understand with this model you pretty much re-ingest data from the "raw" s…

think BI tools, analytics dashboards for exploratory analysis, or even just exploratory analysis on the terminal with it's rich query capabilities. you can keep analytics data in SQLite, but DuckDB will process it faster/easier for the analytics use cases.

Re: DuckDB 0.7.0

#19

Post by [deleted] »

adlpz wrote: post #12
smt88 wrote: post #5

This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?

I'm interested in this, too. I can totally see how not having to manage a standalone RDBMS makes sense. But, what's the real-world advantage over something like SQLite? I mean, the idea of an in-memory relational engine for things like games or embedded totally makes sense, but this seems to target large datasets and deep analysis. As far as I understand with this model you pretty much re-ingest data from the "raw" s…

[deleted]

Re: DuckDB 0.7.0

#20

Post by ZephyrBlu »

colesantiago wrote: post #16
ddorian43 wrote: post #10

Earlier quoted context omitted.

> I don’t understand what it means. It's like Sqlite(OLTP) but for OLAP.

This is still confusing, what do I use this for exactly?

OLAP databases are column oriented and are optimized for querying large amounts of high dimensional data (E.g. many columns). They're usually used for analytics. They don't support some features that OLTP databases have, like transactions.

OLTP databases are your standard database like MySQL, Postgres, etc.

You use an OLAP database if you want to query billions of rows over many different columns. Obviously they can be used for smaller workloads, but I'm exaggerating to show their strengths.

Post reply on HN