Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow 3.0

arrow.apache.org

Post reply on HN
Print view View original

61–70 of 204 posts

Re: Apache Arrow 3.0

#61

Post by mumblemumble »

waynesonfire wrote: post #47
georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.

It's not just a serde. One of its key use cases is eliminating serde.

Re: Apache Arrow 3.0

#62

Post by shafiemukhre »

georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

True. Arrow is awesome and Dremio is using it as well as their built in memory. I tried it and it is increadibily fast. The future of data ecosystem is gonna be amazing

Re: Apache Arrow 3.0

#63

Post by waynesonfire »

TuringTest wrote: post #50
waynesonfire wrote: post #47

Earlier quoted context omitted.

Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.

If it works as a universal intermediate exchange language, it could help standardize connections among disparate systems. When you have N systems, it takes N^2 translators to build direct connections to transfer data between them; but it only takes N translators if all them can talk the same exchange language.

can you define what at translator is? I don't understand the complexity you're constructing. I have N systems and they talk protobuf. What's the problem?

Re: Apache Arrow 3.0

#64

Post by liminal »

Would really love to see first class support for Javascript/Typescript for data visualization purposes. The columnar format would naturally lend itself to an Entity-Component style architecture with TypedArrays.

Re: Apache Arrow 3.0

#65

Post by waynesonfire »

mumblemumble wrote: post #61
waynesonfire wrote: post #47

Earlier quoted context omitted.

Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.

It's not just a serde. One of its key use cases is eliminating serde.

I just don't believe you. My CPU doesn't understand Apache Arrow 3.0.

Re: Apache Arrow 3.0

#66

Post by sethhochberg »

rubicon33 wrote: post #54
georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Wait so you're telling me I could store data as a PDF file, and access it easily / quickly as SQL?

If you found/wrote a adapter to translate your structured PDF into Arrow's format, yes - the idea is that you can wire up anything that can produce Arrow data to anything that can consume Arrow data.

Re: Apache Arrow 3.0

#68

Post by chrisweekly »

waynesonfire wrote: post #47
georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.

For those wondering what a SerDe is: https://docs.serde.rs/serde/

Re: Apache Arrow 3.0

#69

Post by TuringTest »

waynesonfire wrote: post #63
TuringTest wrote: post #50

Earlier quoted context omitted.

If it works as a universal intermediate exchange language, it could help standardize connections among disparate systems. When you have N systems, it takes N^2 translators to build direct connections to transfer data between them; but it only takes N translators if all them can talk the same exchange language.

can you define what at translator is? I don't understand the complexity you're constructing. I have N systems and they talk protobuf. What's the problem?

By a translator, I mean a library that allows accessing data from different subsystems (either languages or OS processes).

In this case, the advantages are that 1) Arrow is language agnostic, so it's likely that it can be used as a native library in your program and 2) it doesn't copy data to make it accessible to another process, so it saves a lot of marshalling / unmarshalling steps (assuming both sides use data in tabular format, which is typical of data analysis contexts).

Re: Apache Arrow 3.0

#70

Post by waynesonfire »

chrisweekly wrote: post #68
waynesonfire wrote: post #47

Earlier quoted context omitted.

Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.

For those wondering what a SerDe is: https://docs.serde.rs/serde/

I meant serialize / deserialize in the literal sense.
Post reply on HN