Apache Arrow is 10 years old
arrow.apache.org
Apache Arrow is 10 years old
1–10 of 74 posts
Re: Apache Arrow is 10 years old
#2I had to look up what Arrow actually does, and I might have to run some performance comparisons vs sqlite.
It's very neat for some types of data to have columns contiguous in memory.
Re: Apache Arrow is 10 years old
#3I had to look up what Arrow actually does, and I might have to run some performance comparisons vs sqlite. It's very neat for some types of data to have columns contiguous in memory.
If I recall, Arrow is more or less a standardized representation in memory of columnar data. It tends to not be used directly I believe, but as the foundation for higher level libraries (like Polars, etc.). That said, I'm not an expert here so might not have full info.
Re: Apache Arrow is 10 years old
#4I had to look up what Arrow actually does, and I might have to run some performance comparisons vs sqlite. It's very neat for some types of data to have columns contiguous in memory.
yeah not necessarily compute (though it has a kernel)!
it's actually many things IPC protocol wire protocol, database connectivity spec etc etc.
in reality it's about an in-memory tabular (columnar) representation that enables zero copy operations b/w languages and engines.
and, imho, it all really comes down to standard data types for columns!
Re: Apache Arrow is 10 years old
#5if I could tell myself in 2015 who had just found the feather library and was using it to power my unhinged topic modeling for power point slides work, and explained what feather would become (arrow) and the impact it would have on the date ecosystem. I would have looked at 2026 me like he was a crazy person.
Yet today I feel it was 2016 dataders who is the crazy one lol
Re: Apache Arrow is 10 years old
#6I had to look up what Arrow actually does, and I might have to run some performance comparisons vs sqlite. It's very neat for some types of data to have columns contiguous in memory.
If I recall, Arrow is more or less a standardized representation in memory of columnar data. It tends to not be used directly I believe, but as the foundation for higher level libraries (like Polars, etc.). That said, I'm not an expert here so might not have full info.
You can absolutely use it directly, but it is painful. The USP of Arrow ist that you can pass bits of memory between Polars, Datafusion, DuckDB, etc. without copying. It's Parquet but for memory.
Re: Apache Arrow is 10 years old
#7I had to look up what Arrow actually does, and I might have to run some performance comparisons vs sqlite. It's very neat for some types of data to have columns contiguous in memory.
>> some performance comparisons vs sqlite.
That's not really the purpose; it's really a language-independent format so that you don't need to change it for say, a dataframe or R. It's columnar because for analytics (where you do lots of aggregations and filtering) this is way more performant; the data is intentionally stored so the target columns are continuous. You probably already know, but the analytics equivalent of SQLite is DuckDB. Arrow can also eliminate the need to serialize/de-serialize data when sharing (ex: a high performance data pipeline) because different consumers / tools / operations can use the same memory representation as-is.
Re: Apache Arrow is 10 years old
#8if I could tell myself in 2015 who had just found the feather library and was using it to power my unhinged topic modeling for power point slides work, and explained what feather would become (arrow) and the impact it would have on the date ecosystem. I would have looked at 2026 me like he was a crazy person. Yet today I feel it was 2016 dataders who is the crazy one lol
Indeed. feather was a library to exchange data between R and pandas dataframes. People tend to bash pandas but its creator (Wes McKinney) has changed the data ecosystem for the better with the learnings coming from pandas.
Re: Apache Arrow is 10 years old
#9Earlier quoted context omitted.
If I recall, Arrow is more or less a standardized representation in memory of columnar data. It tends to not be used directly I believe, but as the foundation for higher level libraries (like Polars, etc.). That said, I'm not an expert here so might not have full info.
You can absolutely use it directly, but it is painful. The USP of Arrow ist that you can pass bits of memory between Polars, Datafusion, DuckDB, etc. without copying. It's Parquet but for memory.
This is true, and as a result IME the problem space is much smaller than Parquet, but it can be really powerful. The reality is most of us don't work in environments where Arrow is needed.
Re: Apache Arrow is 10 years old
#10if I could tell myself in 2015 who had just found the feather library and was using it to power my unhinged topic modeling for power point slides work, and explained what feather would become (arrow) and the impact it would have on the date ecosystem. I would have looked at 2026 me like he was a crazy person. Yet today I feel it was 2016 dataders who is the crazy one lol
Indeed. feather was a library to exchange data between R and pandas dataframes. People tend to bash pandas but its creator (Wes McKinney) has changed the data ecosystem for the better with the learnings coming from pandas.
Do people bash pandas? If so, it reminds me of Bjarne's quip that the two types of programming languages are the ones people complain about and the ones nobody uses.