This is the first time I heard about DuckDB. Why someone would use that instead of a mature engine like SQLite?
Speed and better connectivity.
[deleted]
DuckDB 0.8
61–70 of 101 posts
Re: DuckDB 0.8
#62I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
> why someone would start something in a memory unsafe language these days
You might like what we (Splitgraph) are building with Seafowl [0], a new database which is written in Rust and based on Datafusion and delta-rs [1]. It's optimized for running at the edge and responding to queries via HTTP with cache-friendly semantics.
[1] https://www.splitgraph.com/blog/seafowl-delta-storage-layer
Re: DuckDB 0.8
#63Another cool new feature that's not mentioned in the blog post is function chaining:
https://github.com/duckdb/duckdb/pull/6725
I've been using DuckDB for filtering and post-processing data, specially strings, and this will make writing complex queries easier. By combining nested functions[0] and text functions[1], sometimes I don't even need to go into a Python notebook.
Re: DuckDB 0.8
#64The main use case for me for DuckDB is in Data Lake-related tasks where cloud providers often do not provide great/cost-effective/otherwise limited services. In the case of AWS, repartitioning Parquet files in S3 via Athena CTAS statements in limited to 100 active partitions, which is a bummer to work around. Therefore, I’m using DuckDB with repartitioning queries, because it doesn’t have the 100 partition limit. I w…
We extensively explored this approach for a use-case and dropped it in favour of BigQuery external data query for one core reason: It can't do streaming reads if you have to do any kind of operation on the underlying data, for almost everything it will have to load all the relevant parquet files locally to do the operations. You might have been able to solve this using row partitions on the underlying files beforehand, but it was too late for us.
Also, it actually turned out to be much more expensive to run the operations in cloud run, than relying on BQ (due to the fact it wouldn't stream as we wished). More than anything it gave me renewed appreciation for BQ.
Re: DuckDB 0.8
#65I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
- DuckDB is written in C++ - SQLite is written in C I wouldn't consider any of those written in a memory safe language. Although SQLite has been battle hardened over many years, while DuckDB is a relatively new project. That being said, has been efforts of reimplementing SQLite in a more memory safe language like Rust. e.g. https://github.com/epilys/rsqlite3
C has known memory footguns. C++, due to it's complexity, has footnukes.
Re: DuckDB 0.8
#66I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
A serious and curious question. Are close to the point with LLMs where we can just point to the source of something like DuckDB, and it’s suite of tests and say “rewrite this in Rust with these set of libraries, and make sure all these tests pass”?
Even if not 100% complete, and produces in idiomatic code, could it work?
Re: DuckDB 0.8
#67I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
> why someone would start something in a memory unsafe language these days You might like what we (Splitgraph) are building with Seafowl [0], a new database which is written in Rust and based on Datafusion and delta-rs [1]. It's optimized for running at the edge and responding to queries via HTTP with cache-friendly semantics. [0] https://seafowl.io [1] https://www.splitgraph.com/blog/seafowl-delta-storage-layer
Getting a 5xx error for your site. Using Firefox mobile if that helps
Re: DuckDB 0.8
#68Earlier quoted context omitted.
- DuckDB is written in C++ - SQLite is written in C I wouldn't consider any of those written in a memory safe language. Although SQLite has been battle hardened over many years, while DuckDB is a relatively new project. That being said, has been efforts of reimplementing SQLite in a more memory safe language like Rust. e.g. https://github.com/epilys/rsqlite3
C has known memory footguns. C++, due to it's complexity, has footnukes.
No, I actually think the opposite.
In most C++ environments you will have std::string, STL vectors, unique_ptr, and RAII generally. Cleaning up memory via RAII discipline is standard programming practice. Manual frees are not typical these days. std::string manages its own memory, and isn't vulnerable to the same buffer overflow/null terminator safety issues that C-style strings are.
While in C you will be probably using null-terminated strings and probably your own hand-rolled linked list and vectors. You will not have RAII or destructors, so you will have manual frees all over.
Perhaps the big difference is that due to the nature of the language, C developers on the whole are probably more careful.
Re: DuckDB 0.8
#69I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
A serious and curious question. Are close to the point with LLMs where we can just point to the source of something like DuckDB, and it’s suite of tests and say “rewrite this in Rust with these set of libraries, and make sure all these tests pass”? Even if not 100% complete, and produces in idiomatic code, could it work?
I don’t even have access to regular Claude so can’t confirm this but the 100K token model they released should in theory be able to handle this to a certain degree.
Re: DuckDB 0.8
#70Earlier quoted context omitted.
A serious and curious question. Are close to the point with LLMs where we can just point to the source of something like DuckDB, and it’s suite of tests and say “rewrite this in Rust with these set of libraries, and make sure all these tests pass”? Even if not 100% complete, and produces in idiomatic code, could it work?
I don’t even have access to regular Claude so can’t confirm this but the 100K token model they released should in theory be able to handle this to a certain degree.
Is anyone having luck with this? It seems too "creative".