Why Polars rewrote its Arrow string data type
1–10 of 70 posts
Re: Why Polars rewrote its Arrow string data type
#2I am getting Vietnam flashbacks of all the small incompatibilities between Numpy arrays and Pandas Series
Re: Why Polars rewrote its Arrow string data type
#3Is there a standalone c/c++ implementation of this string type anywhere?
Re: Why Polars rewrote its Arrow string data type
#4Is there a standalone c/c++ implementation of this string type anywhere?
It looks a lot like cedarDB's "german strings".
https://cedardb.com/blog/german_strings/>
You could probably write a C++ implementation based on the article.
Note that it's not all that useful if you don't plan on searching for text based on their prefix. From my understanding, this is mostly a better way to store SStables in RAM/partially on disk if you mmap.
Re: Why Polars rewrote its Arrow string data type
#5I am getting Vietnam flashbacks of all the small incompatibilities between Numpy arrays and Pandas Series
Re: Why Polars rewrote its Arrow string data type
#6I am getting Vietnam flashbacks of all the small incompatibilities between Numpy arrays and Pandas Series
[deleted]
Re: Why Polars rewrote its Arrow string data type
#7I might have missed it in the article but I'm not sure why the prefix is stored for strings that can't be inlined.
Re: Why Polars rewrote its Arrow string data type
#8> “short string optimization”: A short enough string can be stored “in place” [...] An optimization that’s impossible in Rust, by the way ;).
Author is not aware of https://docs.rs/compact_str/latest/compact_str/ or https://github.com/bodil/smartstring
Re: Why Polars rewrote its Arrow string data type
#9I might have missed it in the article but I'm not sure why the prefix is stored for strings that can't be inlined.
Ctrl-F for "Some motivations are as follows" under the "String view with short string optimizations" section here: https://docs.google.com/document/d/12aZi8Inez9L_JCtZ6gi2XDbQ...
Copying here:
> Having the 4-byte prefix directly accessible (without indirection through an offset into a separate data buffer) can substantially improve the performance of comparisons returning false. This prefix can be encoded with multi-column hash keys to accelerate aggregations, joins. Sorts would likely also be significantly faster with this representation (experiments would tell for certain)
> Certain algorithms (for example “prefix of string” or “suffix of string” — e.g. PREFIX(“foobar”, 3) -> “bar”) can execute by manipulating StringView values only and not requiring any memory copying of large strings.
This document was an early proposal for adding what is now called the StringView (and ByteView) types to the Arrow format itself.
Re: Why Polars rewrote its Arrow string data type
#10[deleted]