Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow 3.0

arrow.apache.org

Post reply on HN
Print view View original

71–80 of 204 posts

Re: Apache Arrow 3.0

#71

Post by wesm »

waynesonfire wrote: post #47
georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.

There's no serde by design (aside from inspecting a tiny piece of metadata indicating the location of each constituent block of memory). So data processing algorithms execute directly against the Arrow wire format without any deserialization.

Re: Apache Arrow 3.0

#72

Post by nevi-me »

chrisweekly wrote: post #68
waynesonfire wrote: post #47

Earlier quoted context omitted.

Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.

For those wondering what a SerDe is: https://docs.serde.rs/serde/

The term likely predates the Rust implementation. SerDe is Serializer & Deserializer, which could be any framework or tool that allows the serialization and deserialization of data.

I first came across the concept in Apache Hive.

Re: Apache Arrow 3.0

#73

Post by ianmcook »

didip wrote: post #58
neovintage wrote: post #7

Earlier quoted context omitted.

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…

Question, doesn't Parquet already do that?

From https://arrow.apache.org/faq/: "Parquet files cannot be directly operated on but must be decoded in large chunks... Arrow is an in-memory format meant for direct and efficient use for computational purposes. Arrow data is... laid out in natural format for the CPU, so that data can be accessed at arbitrary places at full speed."

Re: Apache Arrow 3.0

#74

Post by nevi-me »

liminal wrote: post #64

Would really love to see first class support for Javascript/Typescript for data visualization purposes. The columnar format would naturally lend itself to an Entity-Component style architecture with TypedArrays.

Have you seen https://github.com/finos/perspective? Though they removed the JS library a few months ago in favour of a WASM build of the C++ library.

Re: Apache Arrow 3.0

#75

Post by waynesonfire »

wesm wrote: post #71
waynesonfire wrote: post #47

Earlier quoted context omitted.

Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.

There's no serde by design (aside from inspecting a tiny piece of metadata indicating the location of each constituent block of memory). So data processing algorithms execute directly against the Arrow wire format without any deserialization.

Of course there is. There is always deserialization. The data format is most definitely not native to the CPU.

Re: Apache Arrow 3.0

#76

Post by snicker7 »

didip wrote: post #58
neovintage wrote: post #7

Earlier quoted context omitted.

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…

Question, doesn't Parquet already do that?

Yes. But parquet is now based on Apache Arrow.

Re: Apache Arrow 3.0

#77

Post by kristjansson »

waynesonfire wrote: post #65
mumblemumble wrote: post #61

Earlier quoted context omitted.

It's not just a serde. One of its key use cases is eliminating serde.

I just don't believe you. My CPU doesn't understand Apache Arrow 3.0.

Not GP post, but it might have been better stated as 'eliminating serde overhead'. Arrow's RPC serialization [1] is basically Protobuf, with a whole lot of hacks to eliminate copies on both ends of the wire. So it's still 'serde', but markedly more efficient for large blocks of tabular-ish data.

[1]: https://arrow.apache.org/docs/format/Flight.html

Re: Apache Arrow 3.0

#78

Post by IshKebab »

Lucasoato wrote: post #24
IshKebab wrote: post #12

Earlier quoted context omitted.

When you want to process large amounts of in-memory tabular data from different languages. You can save it to disk too using Apache Parquet but I evaluated Parquet and it is very immature. Extremely incomplete documentation and lots of Arrow features are just not supported in Parquet unfortunately.

Do you mean the Parquet format? I don't think Parquet is immature, it is used in so many enterprise environments, it's is one of the few columnar file format for batch analysis and processing. It preforms so well... But I'm curious to know your opinion on this, so feel free to add some context to your position!

Yeah I do. For example Apache Arrow supports in memory compression. But Parquet does not support that. I had to look through the code to find that out, and I found many instances of basically `throw "Not supported"`. And yeah as I said the documentation is just non-existent.

If you are already using Arrow, or you absolutely must use a columnar file format then it's probably a good option.

Re: Apache Arrow 3.0

#79

Post by wesm »

kristjansson wrote: post #77
waynesonfire wrote: post #65

Earlier quoted context omitted.

I just don't believe you. My CPU doesn't understand Apache Arrow 3.0.

Not GP post, but it might have been better stated as 'eliminating serde overhead'. Arrow's RPC serialization [1] is basically Protobuf, with a whole lot of hacks to eliminate copies on both ends of the wire. So it's still 'serde', but markedly more efficient for large blocks of tabular-ish data. [1]: https://arrow.apache.org/docs/format/Flight.html

> Arrow's serialization is Protobuf

Incorrect. Only Arrow Flight embeds the Arrow wire format in a Protocol Buffer, but the Arrow protocol itself does not use Protobuf.

Re: Apache Arrow 3.0

#80

Post by data_ders »

wesm wrote: post #16
georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Microsoft is also on top of this with their Magpie project http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf "A common, efficient serialized and wire format across data engines is a transformational development. Many previous systems and approaches (e.g., [26, 36, 38, 51]) have observed the prohibitive cost of data conversion and transfer, precluding optimizers from exploiting inter-DBMS performance advantages.…

way cool! Is magpie end-user facing anywhere yet? We were using the azureml-dataprep library for a while which seems similar but not all of magpie
Post reply on HN