11–20 of 74 posts
Earlier quoted context omitted.
Indeed. feather was a library to exchange data between R and pandas dataframes. People tend to bash pandas but its creator (Wes McKinney) has changed the data ecosystem for the better with the learnings coming from pandas.
Do people bash pandas? If so, it reminds me of Bjarne's quip that the two types of programming languages are the ones people complain about and the ones nobody uses.
Earlier quoted context omitted.
Do people bash pandas? If so, it reminds me of Bjarne's quip that the two types of programming languages are the ones people complain about and the ones nobody uses.
polars people do - although I wouldn't call polars something that nobody uses.
Earlier quoted context omitted.
polars people do - although I wouldn't call polars something that nobody uses.
I also use polars in new projects. I think Wes McKinney also uses it. If I remember correctly I saw him commenting on some polars memory related issues on GitHub. But a good chunk of polars' success can be attributed to Arrow which McKinney co-created. All the gripes people have with pandas, he had them too and built something powerful to overcome those.
I had to look up what Arrow actually does, and I might have to run some performance comparisons vs sqlite. It's very neat for some types of data to have columns contiguous in memory.
You can also store arrow on disk but it is mainly used as in-memory representation.
I had to look up what Arrow actually does, and I might have to run some performance comparisons vs sqlite. It's very neat for some types of data to have columns contiguous in memory.
>> some performance comparisons vs sqlite. That's not really the purpose; it's really a language-independent format so that you don't need to change it for say, a dataframe or R. It's columnar because for analytics (where you do lots of aggregations and filtering) this is way more performant; the data is intentionally stored so the target columns are continuous. You probably already know, but the analytics equivalent…
if I could tell myself in 2015 who had just found the feather library and was using it to power my unhinged topic modeling for power point slides work, and explained what feather would become (arrow) and the impact it would have on the date ecosystem. I would have looked at 2026 me like he was a crazy person. Yet today I feel it was 2016 dataders who is the crazy one lol
Indeed. feather was a library to exchange data between R and pandas dataframes. People tend to bash pandas but its creator (Wes McKinney) has changed the data ecosystem for the better with the learnings coming from pandas.
Really, prefer DuckDB SQL these days for anything that needs to perform well, and feel like SQL is easier to grok than python code most of the time.
I had to look up what Arrow actually does, and I might have to run some performance comparisons vs sqlite. It's very neat for some types of data to have columns contiguous in memory.
>> some performance comparisons vs sqlite. That's not really the purpose; it's really a language-independent format so that you don't need to change it for say, a dataframe or R. It's columnar because for analytics (where you do lots of aggregations and filtering) this is way more performant; the data is intentionally stored so the target columns are continuous. You probably already know, but the analytics equivalent…
Not sure if I misunderstood, what are the chances those different consumers / tools / operations are running in your memory space?
Earlier quoted context omitted.
>> some performance comparisons vs sqlite. That's not really the purpose; it's really a language-independent format so that you don't need to change it for say, a dataframe or R. It's columnar because for analytics (where you do lots of aggregations and filtering) this is way more performant; the data is intentionally stored so the target columns are continuous. You probably already know, but the analytics equivalent…
> Arrow can also eliminate the need to serialize/de-serialize data when sharing (ex: a high performance data pipeline) because different consumers / tools / operations can use the same memory representation as-is. Not sure if I misunderstood, what are the chances those different consumers / tools / operations are running in your memory space?
You still have to transfer the data, but you remove the need for a transformation before writing to the wire, and a transformation when reading from the wire.