Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB 0.7.0

duckdb.org

Post reply on HN
Print view View original

31–40 of 66 posts

Re: DuckDB 0.7.0

#31

Post by animuchan »

ZephyrBlu wrote: post #23

I haven't used it yet, but DuckDB looks really cool. I'm looking forwards to what MotherDuck releases with it. Having a great local-first product focused on datasets Meta comment: it's fascinating to me that so many people seem to have never heard of OLAP databases.

One of those people here (not that I'm proud of it or anything) — I guess a lot of software engineers just don't do data analysis so heavy it requires a different kind of database.

Not in the utterly pedestrian CRUD apps I write, anyway.

Re: DuckDB 0.7.0

#32

Post by 0x008 »

shanipribadi wrote: post #15
0x008 wrote: post #9

> DuckDB is an in-process SQL OLAP database management system I don’t understand what it means. Can someone explain? I don’t get why they put such a complicated claim with unexplained acronyms on their homepage. When I shop for a db, when should I consider duck DB compared to for example Postgres or MySQL? Or do they compete with arrow or parquet? To me it’s unclear because they don’t say what they compete against.

https://en.wikipedia.org/wiki/Online_analytical_processing as opposed to https://en.wikipedia.org/wiki/Online_transaction_processing DuckDB is when you need to do OLAP analysis, and the data fits in a single node (your laptop), but it's too large for plain excel. technically you can use PG/MySQL/Python+Numpy+Pandas to process those data for that use case as well, but DuckDB does it easier/faster most of the time.

ok That sounds pretty interesting.. i am curious to find out when I would use this over parquet or pandas.

Re: DuckDB 0.7.0

#33

Post by merricksb »

Major discussion of project 2 days ago:

https://news.ycombinator.com/item?id=34741195 (160 points/2 days ago/97 comments)

Also:

https://news.ycombinator.com/item?id=33612898 – DuckDB 0.6.0 (36 points/89 days ago)

https://news.ycombinator.com/item?id=31355050 – Friendlier SQL with DuckDB (366 points/9 months ago/133 comments)

And many others as per dang's comment:

https://news.ycombinator.com/item?id=34746724

Re: DuckDB 0.7.0

#34

Post by santiagobasulto »

chrisjc wrote: post #25
0x008 wrote: post #9

> DuckDB is an in-process SQL OLAP database management system I don’t understand what it means. Can someone explain? I don’t get why they put such a complicated claim with unexplained acronyms on their homepage. When I shop for a db, when should I consider duck DB compared to for example Postgres or MySQL? Or do they compete with arrow or parquet? To me it’s unclear because they don’t say what they compete against.

> in-process think of sqlite. > OLAP think data warehouse. Columnar for analytical workloads. If you want something "in-process" then you're probably going to have to decide between sqlite and duckdb. If your workload is 1) individual fast and frequent read-write operations (OLTP), then you should probably pick sqlite. 2) massive amounts of read-heavy analytical operations (OLAP), then you should probably pick duckdb…

How do you use DuckDB in production for a company? Store SQLite file in something like S3, sync it once per day and run DuckDB with it?

Re: DuckDB 0.7.0

#35

Post by chrisjc »

eliasmacpherson wrote: post #2

So say I wanted to try out a workload on various sql databases, mariadb, sqllite, postgres - is there a database that will act as a front end to them? I find this 'support for pluggable database engines' intriguing. Not least because I can then claim to have used all of the database engines in anger :-) (I know that it is probably a dumb question due to the following, but I asked anyway: https://en.wikipedia.org/wiki…

It's not a dumb question at all. I'm pretty knowledgeable with DBs and still find it very difficult to understand how many of these front-end/pass-through engines work.

Checkout Postgres Foreign Data Wrappers. That might be the most well known approach for accessing one database through another. The Supabase team wrote an interesting piece about this recently.

https://supabase.com/blog/postgres-foreign-data-wrappers-rus...

You might also want to try out duckdb's approach to reading other DBs (or DB files). They talk about how they can "import" a sqlite DB in the above 0.7.0 announcement, but also have some other examples in their duckdblabs github project. Check out their "...-scanner" repos:

https://github.com/duckdblabs/postgres_scanner

https://github.com/duckdblabs/sqlite_scanner

Re: DuckDB 0.7.0

#36

Post by frodowtf »

ZephyrBlu wrote: post #23

I haven't used it yet, but DuckDB looks really cool. I'm looking forwards to what MotherDuck releases with it. Having a great local-first product focused on datasets Meta comment: it's fascinating to me that so many people seem to have never heard of OLAP databases.

It's also fascinating to me that some people seem to believe it's HN's job to define these terms. Seriously, the definition of OLAP is one search away and it is not hard to grasp...

Re: DuckDB 0.7.0

#37

Post by wiredfool »

santiagobasulto wrote: post #34
chrisjc wrote: post #25

Earlier quoted context omitted.

> in-process think of sqlite. > OLAP think data warehouse. Columnar for analytical workloads. If you want something "in-process" then you're probably going to have to decide between sqlite and duckdb. If your workload is 1) individual fast and frequent read-write operations (OLTP), then you should probably pick sqlite. 2) massive amounts of read-heavy analytical operations (OLAP), then you should probably pick duckdb…

How do you use DuckDB in production for a company? Store SQLite file in something like S3, sync it once per day and run DuckDB with it?

More like -- you have a bunch of parquet/csv files in s3 (data lake/house/shore/party/whatever), and duckdb can query them using sql, from python bindings or via a cli.

Re: DuckDB 0.7.0

#38

Post by krishadi »

0x008 wrote: post #9

> DuckDB is an in-process SQL OLAP database management system I don’t understand what it means. Can someone explain? I don’t get why they put such a complicated claim with unexplained acronyms on their homepage. When I shop for a db, when should I consider duck DB compared to for example Postgres or MySQL? Or do they compete with arrow or parquet? To me it’s unclear because they don’t say what they compete against.

If you are a web-developer, building application that store username, email, password and all that stuff, where you'd have to do multiple row level fetch of the data with a lot of parallel connections then you'd use postgres or the like.

If you are a data analyst, analysing a lot of data, which isn't updated in real time, where you'd have to do joins, aggregates which are usually column wise functions, you'd use something like a OLAP db, where duckdb is great for that!

Re: DuckDB 0.7.0

#39

Post by rfoo »

0x008 wrote: post #32
shanipribadi wrote: post #15

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Online_analytical_processing as opposed to https://en.wikipedia.org/wiki/Online_transaction_processing DuckDB is when you need to do OLAP analysis, and the data fits in a single node (your laptop), but it's too large for plain excel. technically you can use PG/MySQL/Python+Numpy+Pandas to process those data for that use case as well, but DuckDB does it easier/faster most of the time.

ok That sounds pretty interesting.. i am curious to find out when I would use this over parquet or pandas.

parquet is just a data file format. Both pandas and DuckDB can query parquet files.

It might be confusing that DuckDB does have its own columnar format, but it's more helpful to think it like "just a query engine (or library) for wrangling tables (or data frames)", i.e. same as pandas.

Re: DuckDB 0.7.0

#40

Post by talolard »

I use Duckdb as a data scientist / analyst. It’s amazing for working with large data locally, because it is very fast and there is almost 0 overhead for use.

For example, I helped an Israeli ngo analyze retailer pricing data (supermarkets must publish prices every day by law). Pandas chokes on data that large, Postgres can handle it but aggregations are very slow. Duckdb is lightning fast.

The traditional alternative I’m familiar with is spark, but it’s such a hassle to setup, expensive to run and not as fast on these kinds of use cases.

I will note that familiarity with Parquet and how columnar engines work is helpful. I have gotten tremendous performance increases when storing the data in a sorted manner in a parquet file, which is ETL overhead.

Still, it’s a very powerful and convenient tool for working with large datasets locally

Post reply on HN