21–30 of 66 posts
This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?
> DuckDB is an in-process SQL OLAP database management system I don’t understand what it means. Can someone explain? I don’t get why they put such a complicated claim with unexplained acronyms on their homepage. When I shop for a db, when should I consider duck DB compared to for example Postgres or MySQL? Or do they compete with arrow or parquet? To me it’s unclear because they don’t say what they compete against.
https://en.wikipedia.org/wiki/Online_analytical_processing as opposed to https://en.wikipedia.org/wiki/Online_transaction_processing DuckDB is when you need to do OLAP analysis, and the data fits in a single node (your laptop), but it's too large for plain excel. technically you can use PG/MySQL/Python+Numpy+Pandas to process those data for that use case as well, but DuckDB does it easier/faster most of the time.
This is an interesting niche. Can anyone explain what they're using it for currently? Much like Redis, I admire the technology but can't think of a project I've worked on that would benefit from it. Is it for games, maybe? Desktop or mobile apps?
> DuckDB is an in-process SQL OLAP database management system I don’t understand what it means. Can someone explain? I don’t get why they put such a complicated claim with unexplained acronyms on their homepage. When I shop for a db, when should I consider duck DB compared to for example Postgres or MySQL? Or do they compete with arrow or parquet? To me it’s unclear because they don’t say what they compete against.
think of sqlite.
> OLAP
think data warehouse. Columnar for analytical workloads.
If you want something "in-process" then you're probably going to have to decide between sqlite and duckdb.
If your workload is
1) individual fast and frequent read-write operations (OLTP), then you should probably pick sqlite.
2) massive amounts of read-heavy analytical operations (OLAP), then you should probably pick duckdb.
Postgres and MySQL are really better suited for out-of-process (shared server) workloads where multiple clients are interacting with the data and resources/compute. They are both row-based OLTP databases, although I do believe Postgres has an option for both table types (HTAP).
Parquet is a file format, just as Avro, JSON, CSV,... are.
Arrow (still grasping this one) is a way that data can be exchanged between systems and processes in such a way that the data is optimized in such a way that doesn't have to go through the extra steps of being shuffled around in memory. For example the data that is returned from a SQL query can be used directly in a Python/Scala/etc dataframe if using Arrow.
I empathize with you about how confusing it all seems, but your curiosity will serve you well. I remember when I was asking these very same questions and it was being led down this road that opened my mind to the world of databases and data engineering.
Google "olap vs oltp" and when you get that, then google "olap vs oltp vs htap".
Or maybe read "The Log: What every software engineer should know about real-time data's unifying abstraction". I know how lame it sounds, but this article really did change the way I think about data.
https://engineering.linkedin.com/distributed-systems/log-wha...
Earlier quoted context omitted.
> I don’t understand what it means. It's like Sqlite(OLTP) but for OLAP.
This is still confusing, what do I use this for exactly?
Really smooth, love it!
> DuckDB is an in-process SQL OLAP database management system I don’t understand what it means. Can someone explain? I don’t get why they put such a complicated claim with unexplained acronyms on their homepage. When I shop for a db, when should I consider duck DB compared to for example Postgres or MySQL? Or do they compete with arrow or parquet? To me it’s unclear because they don’t say what they compete against.
> in-process think of sqlite. > OLAP think data warehouse. Columnar for analytical workloads. If you want something "in-process" then you're probably going to have to decide between sqlite and duckdb. If your workload is 1) individual fast and frequent read-write operations (OLTP), then you should probably pick sqlite. 2) massive amounts of read-heavy analytical operations (OLAP), then you should probably pick duckdb…
Now THAT is easy to understand. Thank you.