31–40 of 70 posts
Don't you lose the in-memory interop with other libraries by doing this? I'm thinking that duckdb will no longer be able to read polars data that has been loaded into memory, as it can currently do, due to duckdb supporting Arrow. Isn't the benefit of arrow that it's supported by many languages and libraries as a standard? Will there be an option to use the "compatible" string format?
> “short string optimization”: A short enough string can be stored “in place” [...] An optimization that’s impossible in Rust, by the way ;). Author is not aware of https://docs.rs/compact_str/latest/compact_str/ or https://github.com/bodil/smartstring
That documentation talks about all the benefits and "can mostly be used as a drop in replacement for String", but what are the tradeoffs? When cannot it be used?
Earlier quoted context omitted.
It seems to be a quote from https://cedardb.com/blog/german_strings/ which is about this German Strings type (implemented in Polars) But yeah, it's pretty ignorant to assume Rust can't do this since the best available examples (as with many things) are in Rust. CompactString is really nice. On a typical modern (64-bit) computer CompactString takes 24 bytes and holds up to 24 bytes of UTF-8 text inline, while also hav…
I remember some C++ colleagues raving on about the standard library having anything and everything they might ever need. Makes sense in a world without sane package managers and package registries, but that mindset just doesn't carry over.
Earlier quoted context omitted.
I remember some C++ colleagues raving on about the standard library having anything and everything they might ever need. Makes sense in a world without sane package managers and package registries, but that mindset just doesn't carry over.
A decent standard library very much makes sense even with sane package managers and registries. Just look at JS. It's awful that you need to hunt for packages for simple stuff (or implement yourself). Stdlib is usually straightforward to use, good quality, trustworthy and good and lasting support
Is there a standalone c/c++ implementation of this string type anywhere?
It looks a lot like cedarDB's "german strings". https://cedardb.com/blog/german_strings/ > You could probably write a C++ implementation based on the article. Note that it's not all that useful if you don't plan on searching for text based on their prefix. From my understanding, this is mostly a better way to store SStables in RAM/partially on disk if you mmap.
Don't you lose the in-memory interop with other libraries by doing this? I'm thinking that duckdb will no longer be able to read polars data that has been loaded into memory, as it can currently do, due to duckdb supporting Arrow. Isn't the benefit of arrow that it's supported by many languages and libraries as a standard? Will there be an option to use the "compatible" string format?
From the article:
> As luck would have it, the Arrow spec was also finally making progress with adding the long anticipated German Style string types to the specification. Which, spoiler alert, is the type we implemented.
Earlier quoted context omitted.
It looks a lot like cedarDB's "german strings". https://cedardb.com/blog/german_strings/ > You could probably write a C++ implementation based on the article. Note that it's not all that useful if you don't plan on searching for text based on their prefix. From my understanding, this is mostly a better way to store SStables in RAM/partially on disk if you mmap.
As an aside, what is "German" about these?
Earlier quoted context omitted.
Why is it impossible in Rust? Do you have any source for that?
It is not. It is not implemented in std::string::String, but (as pointed out elsewhere in this thread) there are other string implementations that have it. It was decided explicitly against for the standard library, because not every optimization is universally good, and keeping String as a thin wrapper over Vec is a good default.
Earlier quoted context omitted.
It is not. It is not implemented in std::string::String, but (as pointed out elsewhere in this thread) there are other string implementations that have it. It was decided explicitly against for the standard library, because not every optimization is universally good, and keeping String as a thin wrapper over Vec is a good default.
When are small strings bad? Parallelism?
You can tear apart these types and reassemble them very easily. For many C++ std types, you cannot do this.