Does anyone know what tool and font was used in the illustrations in the article?
Most likely Excalidraw and Virgil.
Why Polars rewrote its Arrow string data type
41–50 of 70 posts
Re: Why Polars rewrote its Arrow string data type
#42Earlier quoted context omitted.
Was this section removed? I'm not seeing it in the linked post.
It seems to be a quote from https://cedardb.com/blog/german_strings/ which is about this German Strings type (implemented in Polars) But yeah, it's pretty ignorant to assume Rust can't do this since the best available examples (as with many things) are in Rust. CompactString is really nice. On a typical modern (64-bit) computer CompactString takes 24 bytes and holds up to 24 bytes of UTF-8 text inline, while also hav…
> I guess the confusion arises because C++ people tend to assume that anywhere Rust differs from the practice in the C++ community it's a mistake, even though that's often because C++ made the wrong choice?
Funny, I hear that a lot from the Rust folks.
Re: Why Polars rewrote its Arrow string data type
#43Earlier quoted context omitted.
It is not. It is not implemented in std::string::String, but (as pointed out elsewhere in this thread) there are other string implementations that have it. It was decided explicitly against for the standard library, because not every optimization is universally good, and keeping String as a thin wrapper over Vec is a good default.
When are small strings bad? Parallelism?
You'll need to branch on most operations to check "am i small?". This may cause issues with the Branche predictor.
Re: Why Polars rewrote its Arrow string data type
#44Don't you lose the in-memory interop with other libraries by doing this? I'm thinking that duckdb will no longer be able to read polars data that has been loaded into memory, as it can currently do, due to duckdb supporting Arrow. Isn't the benefit of arrow that it's supported by many languages and libraries as a standard? Will there be an option to use the "compatible" string format?
DuckDB has their own string type (quite similar to this) that deviates from Arrow (large)-string type, so it had to do a copy anyway. Nothing has changed on that front.
Re: Why Polars rewrote its Arrow string data type
#45> As I mentioned above already pre-allocating the required size of data is hard. This leads to many reallocations and memcopy’s during the building of this type.
Well. Reallocations have to happen mostly because the virtual memory space is flat, so you can't just grow your allocations without the possibility to accidentally bumping into some other object. But having non-flat virtual memory space is really inconvenient (Segment selectors! CHERI! And what about muh address arithmetic?) for other reasons, so here we are.
I toyed with the idea of having a specialized memory allocator for incrementally growing, potentially very large buffers by having it space allocations by, say, 16 GiB, and then there would be the "finalize" operation which would hand over the buffer's contents to malloc by asking malloc to allocate the exact final buffer size (rounded up to the page size) and then, instead of memcpy-ing the data, I'd persuade the OS to remap the physical pages of the existing allocation into the virtual address returned by malloc. The original buffer's virtual addresses then would become unmapped and could be reused.
Unfortunately, I couldn't quite persuade the OS to do that with user-available memory management API so it all came to nothing. I believe there was similar research in the early 90s and it failed because it too required custom OS modifications.
Re: Why Polars rewrote its Arrow string data type
#46Earlier quoted context omitted.
I remember some C++ colleagues raving on about the standard library having anything and everything they might ever need. Makes sense in a world without sane package managers and package registries, but that mindset just doesn't carry over.
A decent standard library very much makes sense even with sane package managers and registries. Just look at JS. It's awful that you need to hunt for packages for simple stuff (or implement yourself). Stdlib is usually straightforward to use, good quality, trustworthy and good and lasting support
I attribute Go's success to a rich OOTB standard library. You can build so much before you reach for third party packages. Heck, for web services, the built-in + their templating gets you pretty far.
Re: Why Polars rewrote its Arrow string data type
#47Re: Why Polars rewrote its Arrow string data type
#48Earlier quoted context omitted.
It seems to be a quote from https://cedardb.com/blog/german_strings/ which is about this German Strings type (implemented in Polars) But yeah, it's pretty ignorant to assume Rust can't do this since the best available examples (as with many things) are in Rust. CompactString is really nice. On a typical modern (64-bit) computer CompactString takes 24 bytes and holds up to 24 bytes of UTF-8 text inline, while also hav…
I remember some C++ colleagues raving on about the standard library having anything and everything they might ever need. Makes sense in a world without sane package managers and package registries, but that mindset just doesn't carry over.
That's a bit silly, the C++ standard library is pretty small ("the intersection of your needs, not the union" is what they say), and it's often not even really good at the few things that it does. Whether that's core stuff like hashmaps and iostreams, or more niche stuff like std::regex.
Re: Why Polars rewrote its Arrow string data type
#49Earlier quoted context omitted.
Why is it impossible in Rust? Do you have any source for that?
It is not. It is not implemented in std::string::String, but (as pointed out elsewhere in this thread) there are other string implementations that have it. It was decided explicitly against for the standard library, because not every optimization is universally good, and keeping String as a thin wrapper over Vec is a good default.
This is one of those things that I wish people would speak more carefully about. I've seen it in every programming language community I've participated in, so it's not a language-specific thing, but... one should not say "language X does not do a thing" when they mean "language X's standard library does not do a thing". That the "language" doesn't do a thing should be reserved for the cases where the language itself really does preclude some particular thing for some reason. Otherwise the relatively inexperienced programmers end up coming away with some really weird mental models of what programming languages can and can not do, just like here. Of course Rust qua Rust can store strings like this, it's basically built for things like this. Any mental model of Rust that thinks Rust qua Rust can't do this is a weird mental model of Rust.
Re: Why Polars rewrote its Arrow string data type
#50Earlier quoted context omitted.
It is not. It is not implemented in std::string::String, but (as pointed out elsewhere in this thread) there are other string implementations that have it. It was decided explicitly against for the standard library, because not every optimization is universally good, and keeping String as a thin wrapper over Vec is a good default.
When are small strings bad? Parallelism?
Rust APIs use string slices (&str) extensively. With its current design, converting from a String to a &str is a no-op; if String instead did small string optimization, converting to &str wouldn't be free. Furthermore, thanks to the borrow checker, Rust code tends to avoid copying strings around, so the benefit of SSO is reduced. C++ does more string copying and didn't have a standard string_view for a long time, so considering the tradeoffs both languages made reasonable decisions.