DuckDB – An in-process SQL OLAP database management system
1–10 of 104 posts
Re: DuckDB – An in-process SQL OLAP database management system
#2DuckDB is awesome!
Last year I was working on something using SQLite, users could perform analytical queries that would scan the entire 3gb db and generate aggregates. It would take at least 45 seconds to do the queries.
I did a dump of the db and imported to DuckDB. The same queries now only take 1.5 seconds with exactly the same SQL.
Obviously there is a trade off, inserts are slower on DuckDB. But for a low write, analytical read app it's perfect.
I tried to use the SQLite connector but struggled to get it working. Need to circle back and have another go.
With their SQLite and Postgres connectors, as a Django dev I would love an app that lets you run specific queries on your DB via DuckDB almost transparently. Would be awesome for analytical dashboards.
Re: DuckDB – An in-process SQL OLAP database management system
#3DuckCon 2023 - State of the Duck - Mark Raasveldt and Hannes Mühleisen
Re: DuckDB – An in-process SQL OLAP database management system
#4I wonder if we'll in the near future also see JVM based in-memory or hybrid OLAP database systems, which will make use of SIMD instructions and columnar storage layouts with the incubating Vector API.
It would be also interesting to see how we can process semi-structured data in a similar way.
Re: DuckDB – An in-process SQL OLAP database management system
#5DuckDB is awesome! Last year I was working on something using SQLite, users could perform analytical queries that would scan the entire 3gb db and generate aggregates. It would take at least 45 seconds to do the queries. I did a dump of the db and imported to DuckDB. The same queries now only take 1.5 seconds with exactly the same SQL. Obviously there is a trade off, inserts are slower on DuckDB. But for a low write,…
Yeah, I see it as the sqlite of the reporting world. Especially with columnar stores and parquet support.
Re: DuckDB – An in-process SQL OLAP database management system
#6I'm duck-curious.
Looking at how it's deployed, as an in process database, how do people actually use this in production? Trying to figure out where I might actually want to think about replacing current databases or analyses with DuckDB.
EG if you deployed new code
1. Do you have a stateful machine you're doing an old school "Kill the old process, start the new process" deploy, and there's some duckdb file on disk that is maintained?
2. Or do you back that duckdb file in some sort of shared disk (Eg EBS), and have a rolling deploy where multiple applications access the same DB at the same time?
3. Or is DuckDB is treated as ephemeral, and you're using it to process data on the fly, so persisted state isn't an issue?
Re: DuckDB – An in-process SQL OLAP database management system
#7I'm duck-curious. Looking at how it's deployed, as an in process database, how do people actually use this in production? Trying to figure out where I might actually want to think about replacing current databases or analyses with DuckDB. EG if you deployed new code 1. Do you have a stateful machine you're doing an old school "Kill the old process, start the new process" deploy, and there's some duckdb file on disk t…
Good questions, I'm thinking about evaluating it along with pola.rs as data science tools. Would be interesting to hear from someone using it already.
Re: DuckDB – An in-process SQL OLAP database management system
#8DuckDB is awesome! Last year I was working on something using SQLite, users could perform analytical queries that would scan the entire 3gb db and generate aggregates. It would take at least 45 seconds to do the queries. I did a dump of the db and imported to DuckDB. The same queries now only take 1.5 seconds with exactly the same SQL. Obviously there is a trade off, inserts are slower on DuckDB. But for a low write,…
I'd need a MySQL/MariaDB connector for it to be really useful for me. But still, it looks great.
Re: DuckDB – An in-process SQL OLAP database management system
#9I'm duck-curious. Looking at how it's deployed, as an in process database, how do people actually use this in production? Trying to figure out where I might actually want to think about replacing current databases or analyses with DuckDB. EG if you deployed new code 1. Do you have a stateful machine you're doing an old school "Kill the old process, start the new process" deploy, and there's some duckdb file on disk t…
Yes using it in production - Stateless and ephemeral. For sure there’s a learning curve.
Re: DuckDB – An in-process SQL OLAP database management system
#10I'm duck-curious. Looking at how it's deployed, as an in process database, how do people actually use this in production? Trying to figure out where I might actually want to think about replacing current databases or analyses with DuckDB. EG if you deployed new code 1. Do you have a stateful machine you're doing an old school "Kill the old process, start the new process" deploy, and there's some duckdb file on disk t…
Yes using it in production - Stateless and ephemeral. For sure there’s a learning curve.
I see. Would it be fair to say you treat it almost like Pandas, except that it has a lower memory footprint since data is written to disk instead of memory. IE you use it for on the fly analysis of large frames of data, not like more traditional database/datawarehouse?