Such an amazing project. Once the geospatial extension lands I'm going to seriously consider switching our geoparquet workflows over.
You mean this one? DuckDB Spatial Extension https://duckdb.org/2023/04/28/spatial.html
Yes, it's still very alpha
DuckDB 0.8
91–100 of 101 posts
Re: DuckDB 0.8
#92Earlier quoted context omitted.
I find clickhouse equally simple. One command to install and then you can either read from local/remote files, create tables, etc
Clickhouse looks more complicated because instead of create table bla ( id number, xyz varchar ) you're doing things like create table bla ( id uint64 CODEC(DoubleDelta(8)), xyz Nullable(LowCardinality(Varchar)) CODEC(ZSTD(1)) ) engine=MergeTree order by id but I do wonder what is the performance difference between actually carefully specifying everything and leaving defaults (except for order by I guess).
Fair enough. Some of the settings will have performance impact, e.g. the low cardinality type, data skipping indexes, etc. and you have to spend some time to learn and experiment yourself. However it’s a powerful db and even the default settings will work well in the majority of cases.
Re: DuckDB 0.8
#93Earlier quoted context omitted.
I find clickhouse equally simple. One command to install and then you can either read from local/remote files, create tables, etc
Clickhouse looks more complicated because instead of create table bla ( id number, xyz varchar ) you're doing things like create table bla ( id uint64 CODEC(DoubleDelta(8)), xyz Nullable(LowCardinality(Varchar)) CODEC(ZSTD(1)) ) engine=MergeTree order by id but I do wonder what is the performance difference between actually carefully specifying everything and leaving defaults (except for order by I guess).
Just leave everything by default - no need specifying any codecs, etc. People often try to show up their knowledge by using advanced features, but that's not good.
Re: DuckDB 0.8
#94The main use case for me for DuckDB is in Data Lake-related tasks where cloud providers often do not provide great/cost-effective/otherwise limited services. In the case of AWS, repartitioning Parquet files in S3 via Athena CTAS statements in limited to 100 active partitions, which is a bummer to work around. Therefore, I’m using DuckDB with repartitioning queries, because it doesn’t have the 100 partition limit. I w…
We extensively explored this approach for a use-case and dropped it in favour of BigQuery external data query for one core reason: It can't do streaming reads if you have to do any kind of operation on the underlying data, for almost everything it will have to load all the relevant parquet files locally to do the operations. You might have been able to solve this using row partitions on the underlying files beforehan…
I don't think this is correct, or I didn't really understand your use case. Could you give an example for "any kind of operation on the underlying data"?
Re: DuckDB 0.8
#95I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
DuckDB is a great piece of software if you are
If you are looking for a query engine implemented in a safe language (Rust) I definitely suggest checking out DataFusion. It is comparable to DuckDB in performance, has all the standard built in SQL functionality, and is extensible in pretty much all areas (query language, data formats, catalogs, user defined functions, etc)
https://arrow.apache.org/datafusion/
Disclaimer I am a maintainer of DataFusion
Re: DuckDB 0.8
#96I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
It is open source, you can do it with whatever you want, like fix it or write it in rust. I did not see the point of complaining it doesn’t work for you because it is not in rust and not doing it.
Re: DuckDB 0.8
#97Earlier quoted context omitted.
- DuckDB is written in C++ - SQLite is written in C I wouldn't consider any of those written in a memory safe language. Although SQLite has been battle hardened over many years, while DuckDB is a relatively new project. That being said, has been efforts of reimplementing SQLite in a more memory safe language like Rust. e.g. https://github.com/epilys/rsqlite3
At the level of engineering of SQLite, the choice of language is almost immaterial. Suggesting a low effort transpilation is a competitive peer seems unserious and vaguely disrespectful.
The CVE list would dispute that assertion. There's a reason Microsoft is rewriting parts of the Windows kernel in Rust, and it isn't trendiness or because the kernel is at a trivial level of engineering.
It's the same reason Torvalds refused to have C++ anywhere near the Linux kernel but is not accepting patches in Rust. The advantage of C is its transparency and simplicity, but its safety has always been a thorn in the industry's side.
RiiR has become a bit of a parody of itself, but there is a large grain of truth from where that sentiment was born.
Life is too short for segfaults and buffer overruns.
Re: DuckDB 0.8
#98Earlier quoted context omitted.
C has known memory footguns. C++, due to it's complexity, has footnukes.
No, I actually think the opposite. In most C++ environments you will have std::string, STL vectors, unique_ptr, and RAII generally. Cleaning up memory via RAII discipline is standard programming practice. Manual frees are not typical these days. std::string manages its own memory, and isn't vulnerable to the same buffer overflow/null terminator safety issues that C-style strings are. While in C you will be probably u…
I started coding professionally in C++ back around 2000. Many things have improved in C++ such as the items you list above, but C++ remains a viciously complicated language that requires constant vigilance while coding in it, especially when more than one thread is involved.
CloudFlare is not lacking in good engineers with tons of both C and C++ experience. They still chose Rust for their replacement of Nginx. Now their crashes are so few and far between, they uncover kernel bugs rather than app-level bugs.
https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
> Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code.
I have never heard anything close to that level of reliability from a C or C++ codebase. Never. And I've worked with truly great programmers before on modern C++ projects. C++ may not have limits, but humans writing C++ provably do.
Re: DuckDB 0.8
#99I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
> I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite instead. I am not sure that you realize that SQLite is written entirely in C -- a quintessential memory unsafe language. I guess quality of software depends on many things b…
SQLite also lives under a literal mountain of automated tests, an engineering effort I'm not sure I've ever seen elsewhere. The library code is absolutely dwarfed by the test code.
...and CVEs still pop up occasionally. The point about memory safety languages still holds, but can be mostly muted given you throw enough tests at the problem.
Re: DuckDB 0.8
#100Earlier quoted context omitted.
No, I actually think the opposite. In most C++ environments you will have std::string, STL vectors, unique_ptr, and RAII generally. Cleaning up memory via RAII discipline is standard programming practice. Manual frees are not typical these days. std::string manages its own memory, and isn't vulnerable to the same buffer overflow/null terminator safety issues that C-style strings are. While in C you will be probably u…
I started coding professionally in C++ back around 2000. Many things have improved in C++ such as the items you list above, but C++ remains a viciously complicated language that requires constant vigilance while coding in it, especially when more than one thread is involved. CloudFlare is not lacking in good engineers with tons of both C and C++ experience. They still chose Rust for their replacement of Nginx. Now th…
I'm not sure who you're arguing against, but it's not me, or it's off topic completely. The discussion and my reply was not between C/C++ and Rust, but between C and C++.
I'm a full-time Rust developer FWIW. But I also did C++ for 10 years prior and worked in the language on and off since the mid-90s. Nobody is arguing in this sub-thread about Rust vs C++, nor am I interested in getting into your religious war.