Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow 3.0

arrow.apache.org

Post reply on HN
Print view View original

51–60 of 204 posts

Re: Apache Arrow 3.0

#51

Post by waynesonfire »

ianmcook wrote: post #48
waynesonfire wrote: post #38

Earlier quoted context omitted.

> It also has the added benefit of eliminating serialization and deserialization of data between processes Is that accurate? It still has to deserialize from apache arrow format to whatever the cpu understands.

The Arrow Feather format is an on-disk representation of Arrow memory. To read a Feather file, Arrow just copies it byte for byte from disk into memory. Or Arrow can memory-map a Feather file so you can operate on it without reading the whole file into memory.

That's exactly how I read every data format.

The advantage you describe is in the operations that can performed against the data. It would be nice to see what this API looks like and how it compares to flatbuffers / pq.

To help me understand this benefit, can you talk through what it's like to add 1 to each record and write it back to disk?

Re: Apache Arrow 3.0

#52

Post by hawk_ »

arjunnarayan wrote: post #23
pindab0ter wrote: post #19

Earlier quoted context omitted.

I hope you don’t mind me asking dumb questions, but how does this differ from the role that say Protocol Buffers fills? To my ears they both facilitate data exchange. Are they comparable in that sense?

protobufs still get encoded and decoded by each client when loaded into memory. arrow is a little bit more like "flatbuffers, but designed for common data-intensive columnar access patterns"

is that at the cost of the ability to do schema evolution?

Re: Apache Arrow 3.0

#53

Post by zten »

waynesonfire wrote: post #38
tomnipotent wrote: post #33

Earlier quoted context omitted.

This is intended for analytical workloads where you're often doing things that can benefit from vectorization (like SIMD). It's much faster to SUM(X) when all values of X are neatly laid out in-memory. It also has the added benefit of eliminating serialization and deserialization of data between processes - a Python process can now write to memory which is read by a C++ process that's doing windowed aggregations, whi…

> It also has the added benefit of eliminating serialization and deserialization of data between processes Is that accurate? It still has to deserialize from apache arrow format to whatever the cpu understands.

The important part to focus on is _between processes_.

Consider Spark and PySpark. The Python bits of Spark are in a sidecar process to the JVM running Spark. If you ask PySpark to create a DataFrame from Parquet data, it'll instruct the Java process to load the data. Its in-memory form will be Arrow. Now, if you want to manipulate that data in PySpark using Python-only libraries, prior to the adoption of Arrow it used to serialize and deserialize the data between processes on the same host. With Arrow, this process is simplified -- however, I'm not sure if it's simplified by exchanging bytes that don't require serialization/deserialization between the processes or by literally sharing memory between the processes. The docs do mention zero-copied shared memory.

Re: Apache Arrow 3.0

#54

Post by rubicon33 »

georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Wait so you're telling me I could store data as a PDF file, and access it easily / quickly as SQL?

Re: Apache Arrow 3.0

#55

Post by [deleted] »

waynesonfire wrote: post #38
tomnipotent wrote: post #33

Earlier quoted context omitted.

This is intended for analytical workloads where you're often doing things that can benefit from vectorization (like SIMD). It's much faster to SUM(X) when all values of X are neatly laid out in-memory. It also has the added benefit of eliminating serialization and deserialization of data between processes - a Python process can now write to memory which is read by a C++ process that's doing windowed aggregations, whi…

> It also has the added benefit of eliminating serialization and deserialization of data between processes Is that accurate? It still has to deserialize from apache arrow format to whatever the cpu understands.

[deleted]

Re: Apache Arrow 3.0

#56

Post by jeffbee »

BenoitP wrote: post #29
neovintage wrote: post #7

Earlier quoted context omitted.

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…

> most of the compute time spent is in serializing and deserializing data. This is to be viewed in light how hardware evolves now. CPU compute power is no longer growing as much (at least for individual cores). But one thing that's still doubling on a regular basis is memory capacity of all kinds (RAM, SSD, etc) and bandwidth of all kinds (PCIe lanes, networking, etc). This divide is getting large and will only conti…

An interesting perspective on the future of computer architecture but it doesn't align well with my experience. CPUs are easier to build and although a lot of ink has been spilled about the end of Moore's Law, it remains the case that we are still on Moore's curve for number of transistors, and since about 15 years ago we are now also on the same slope for # of cores per CPU. We also still enjoy increasing single-thread performance, even if not at the rates of past innovation.

DRAM, by contrast, is currently stuck. We need materials science breakthroughs to get beyond the capacitor aspect ratio challenge. RAM is still cheap but as a systems architect you should get used to the idea that the amount of DRAM per core will fall in the future, by amounts that might surprise you.

Re: Apache Arrow 3.0

#57

Post by cornstalks »

rcoveson wrote: post #25
pindab0ter wrote: post #19

Earlier quoted context omitted.

I hope you don’t mind me asking dumb questions, but how does this differ from the role that say Protocol Buffers fills? To my ears they both facilitate data exchange. Are they comparable in that sense?

Better to compare it to Cap'n Proto instead. Arrow data is already laid out in a usable way. For example, an Arrow column of int64s is an 8-byte aligned memory region of size 8*N bytes (plus a bit vector for nullity), ready for random access or vectorized operations. Protobuf, on the other hand, would encode those values as variable-width integers. This saves a lot of space, which might be better for transfer over a…

> Think of Arrow as standardized shared memory using struct-of-arrays layout, Cap'n Proto as standardized shared memory using array-of-structs layout

I just want to say thank you for this part of the sentence. I understand struct-of-arrays vs array-of-structs, and now I finally understand what the heck Arrow is.

Re: Apache Arrow 3.0

#58

Post by didip »

neovintage wrote: post #7
mushufasa wrote: post #3

Can someone ELI5 what problems are best solved by apache arrow?

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…

Question, doesn't Parquet already do that?

Re: Apache Arrow 3.0

#59

Post by jedbrown »

BenoitP wrote: post #29
neovintage wrote: post #7

Earlier quoted context omitted.

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…

> most of the compute time spent is in serializing and deserializing data. This is to be viewed in light how hardware evolves now. CPU compute power is no longer growing as much (at least for individual cores). But one thing that's still doubling on a regular basis is memory capacity of all kinds (RAM, SSD, etc) and bandwidth of all kinds (PCIe lanes, networking, etc). This divide is getting large and will only conti…

This is backward -- this sort of serialization is overwhelmingly bottlenecked on bandwidth (not CPU). (Multi-core) compute improvements have been outpacing bandwidth improvements for decades and have not stopped. Serialization is a bottleneck because compute is fast/cheap and bandwidth is precious. This is also reflected in the relative energy to move bytes being increasingly larger than the energy to do some arithmetic on those bytes.

Re: Apache Arrow 3.0

#60

Post by georgewfraser »

jayd16 wrote: post #30

Can someone dig into the pros and cons of the columnar aspect of Arrow? To some degree there are many other data transfer formats but this one seems to promote its columnar orientation. Things like eg. protobuffers support hierarchical data which seems like a superset of columns. Is there a benefit to a column based format? Is it an enforced simplification to ensure greater compatibility or is there some other reason…

The main benefit of a columnar representation in memory is it's more cache friendly for a typical analytical workload. For example, if I have a dataframe:

  (A int, B int, C int, D int)
And I write:

  A + B
In a columnar representation, all the As are next to each other, and all the Bs are next to each other, so the process of (A and B in memory) => (A and B in CPU registers) => (addition) => (A + B result back to memory) will be a lot more efficient.

In a row-oriented representation like protobuf, all your C and D values are going to get dragged into the CPU registers alongside the A and B values that you actually want.

Column-oriented representation is also more friendly to SIMD CPU instructions. You can still use SIMD with a row-oriented representation, but you have to use gather-scatter operations which makes the whole thing less efficient.

Post reply on HN