This is the first time I heard about DuckDB. Why someone would use that instead of a mature engine like SQLite?
Speed and better connectivity.
DuckDB 0.8
51–60 of 101 posts
Re: DuckDB 0.8
#52Setting up SQL, creating a well-designed schema, and optimizing queries can be a complex, multi-step process. Databases often feel like black boxes, requiring significant training and engineering to get right. One needs to tweak both queries, schema and database configuration to get best results. So why do we need SQL? I know NoSQL came and went (was it nosql or no-relations? not sure...) but honestly I can't think o…
I found this comment in a previous discussion insightful (https://news.ycombinator.com/item?id=34580448)
This simple command `UPDATE users SET preference = 'blue' WHERE id = 123` virtually contains:
* concurrency control
* statistics, which will help:
* execution plans evaluation, which will reduce IO cost with the help of:
* (several categories of) indexes
* type checking
* data invariants checking
* point in time recovery
* enables dataset-wide backup strategies
* ISO standard way of structuring data
* an interface that empowers business people
And that's just a few items off the top of my mind.
Re: DuckDB 0.8
#53I used duckdb successfully in prod to replace SQL Server. We have a micro batch that generates around 5 billion rows of very wide tables every 3 minutes. These data used to go into SQL server, only to be replaced by the new batch of 5 billions and gets marked for deletion. SQL Server was struggling with all these purge activities. Replacing with DuckDB made things much lighter and faster. The only issue I faced is th…
I'm curious, what kind of problem needs 5 billion rows every 3 minutes to replace the previous 5 billion rows? By "micro batch" I understand batch processing of some kind?
financial risk calculations with many sensitivities
Re: DuckDB 0.8
#54The feature to register arbitrary Python functions as scalar functions within SQL queries is pretty insane. Writing custom functions to manipulate data in ways that aren't possible with SQL, basically without fetching all of it at once in memory and then applying. I hope thats the case
SQLite has similar functionality.
e.g PHP
Re: DuckDB 0.8
#55I've been thinking about rebuilding the website analytics service I have, to be around sharded duckdb and just giving people raw SQL access. ClickHouse is the alternative, but this approach has some interesting advantages, like simplicity.
I find clickhouse equally simple. One command to install and then you can either read from local/remote files, create tables, etc
Re: DuckDB 0.8
#56Such an amazing project. Once the geospatial extension lands I'm going to seriously consider switching our geoparquet workflows over.
I'm just minorly annoyed that they're not calling it GeoDuck.
Re: DuckDB 0.8
#57Earlier quoted context omitted.
- DuckDB is written in C++ - SQLite is written in C I wouldn't consider any of those written in a memory safe language. Although SQLite has been battle hardened over many years, while DuckDB is a relatively new project. That being said, has been efforts of reimplementing SQLite in a more memory safe language like Rust. e.g. https://github.com/epilys/rsqlite3
At the level of engineering of SQLite, the choice of language is almost immaterial. Suggesting a low effort transpilation is a competitive peer seems unserious and vaguely disrespectful.
>Finally, one of the best written software paired with one of the best writable programming language‽ Fearless and memory safe, since the uncountable amount of unsafe {} blocks makes you not care anymore.
Plus it seems project is a parody of the RiiR trend.
Re: DuckDB 0.8
#58Damn. This database quacks. Huge list of awesome super helpful tools. Tons of great features & speed for everyone. Also love the stuff from the edge, such as Arrow DataBase Connector support, > From this release, DuckDB natively supports ADBC. We’re happy to be one of the first systems to offer native support, and DuckDB’s in-process design fits nicely with ADBC.
ADBC is going to be huge for BI tooling. Right now, most of them go through ODBC/JDBC which are row-oriented formats. So your columnar data is always reshaped from column-oriented (in your data warehouse / lakehouse) to row-oriented and often back again. The data is also serialized, deserialized on top of that.
We got 8-20X speed-up replacing JDBC with Arrow Flight Service, which is a more limited version of ADBC - https://www.youtube.com/watch?v=nCxIpMXvCp0
Re: DuckDB 0.8
#59I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
I remember chasing memory bugs in C# and Java projects at work. These are usually considered memory-safe languages yet their sophisticated garbage collectors are not panacea. There is a reason things like this https://docs.oracle.com/javase/8/docs/api/java/lang/ref/Weak... exist. Or why you might re-use existing objects in certain situations https://www.oreilly.com/library/view/java-performance-tuning... . On the oth…
Memory leaks are annoying and, yes, you can get them in memory safe languages.
But they are way less severe than memory corruption. Memory unsafe languages are liable to undefined behaviour, which is actively dangerous, both in theory and practice.
Re: DuckDB 0.8
#60The main use case for me for DuckDB is in Data Lake-related tasks where cloud providers often do not provide great/cost-effective/otherwise limited services.
In the case of AWS, repartitioning Parquet files in S3 via Athena CTAS statements in limited to 100 active partitions, which is a bummer to work around. Therefore, I’m using DuckDB with repartitioning queries, because it doesn’t have the 100 partition limit.
I wrote a blog post about it at https://tobilg.com/casual-data-engineering-or-a-poor-mans-da... Additionally, to get started with using DuckDB serverlessly in Lambda functions, you can have a look at https://tobilg.com/using-duckdb-in-aws-lambda