Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow 3.0

arrow.apache.org

Post reply on HN
Print view View original

111–120 of 204 posts

Re: Apache Arrow 3.0

#111

Post by nomel »

georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases.

Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data.

[1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...

Re: Apache Arrow 3.0

#112

Post by cdash »

poorman wrote: post #44
neovintage wrote: post #7

Earlier quoted context omitted.

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…

Not only in between processes, but also in between languages in a single process. In this POC I spun up a Python interpreter in a Go process and pass the Arrow data buffer between processes in constant time. https://github.com/nickpoorman/go-py-arrow-bridge

How is this any better than something like flatbuffers though?

Re: Apache Arrow 3.0

#114

Post by dlnovell »

waynesonfire wrote: post #94
wesm wrote: post #81

Earlier quoted context omitted.

I challenge you to have a closer look at the project. Deserialization by definition requires bytes or bits to be relocated from their position in the wire protocol to other data structures which are used for processing. Arrow does not require any bytes or bits to be relocated. So if a "C array of doubles" is not native to the CPU, then I don't know what is.

CPUs come in many flavors. One area where they differ is in the way that bytes of a word are represented in memory. Two common formats are Big Endian and Little Endian. This is an example where a "C array of doubles" would be incompatible and some form of deserilaziation would be needed. My understanding is that an apache arrow library provides an API to manipulate the format in a platform agnostic way. But to claim…

I wish I had your confidence, to argue with Wes McKinney about the details of how Arrow works

Re: Apache Arrow 3.0

#118

Post by archagon »

For use as a file format, where one priority is to compress columnar data as well as possible, the practical difference between Arrow (via Feather?), Parquet, and ORC is still somewhat vague to me. During my last investigation, I got the impression that Arrow worked great as a standard, interoperable, in-memory columnar format, but didn't compress nearly as well as ORC or Parquet due to lack of RLE and other compression schemes (other than dictionary). Is this still the case? Is there a world where Arrow completely supplants Parquet and/or ORC?

EDIT: Just found https://wesmckinney.com/blog/arrow-columnar-abadi, which helps answer this question.

Re: Apache Arrow 3.0

#120

Post by ska »

waynesonfire wrote: post #94
wesm wrote: post #81

Earlier quoted context omitted.

I challenge you to have a closer look at the project. Deserialization by definition requires bytes or bits to be relocated from their position in the wire protocol to other data structures which are used for processing. Arrow does not require any bytes or bits to be relocated. So if a "C array of doubles" is not native to the CPU, then I don't know what is.

CPUs come in many flavors. One area where they differ is in the way that bytes of a word are represented in memory. Two common formats are Big Endian and Little Endian. This is an example where a "C array of doubles" would be incompatible and some form of deserilaziation would be needed. My understanding is that an apache arrow library provides an API to manipulate the format in a platform agnostic way. But to claim…

You are right that if you want to do this in a heterogeneous computing environment, one of the layouts is going to be "wrong" and require an extra step regardless of how you do this.

But ... (a) this is way less common than it was decades ago (rare use cases we are talking about here ) and (b) it seems to be addressed in a sensible way (i.e. Arrow defaults to little-endian, but you could swap it on a big-endian network). I think it includes utility functions for conversion also.

So the usual case incurs no overhead, and the corner cases are covered. I'm not sure exactly what you are complaining about, unless it's the lack of liberally sprinkling ("no deserialization in most use cases") or whatever around the comments?

Post reply on HN