Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow 3.0

arrow.apache.org

Post reply on HN
Print view View original

81–90 of 204 posts

Re: Apache Arrow 3.0

#81

Post by wesm »

waynesonfire wrote: post #75
wesm wrote: post #71

Earlier quoted context omitted.

There's no serde by design (aside from inspecting a tiny piece of metadata indicating the location of each constituent block of memory). So data processing algorithms execute directly against the Arrow wire format without any deserialization.

Of course there is. There is always deserialization. The data format is most definitely not native to the CPU.

I challenge you to have a closer look at the project.

Deserialization by definition requires bytes or bits to be relocated from their position in the wire protocol to other data structures which are used for processing. Arrow does not require any bytes or bits to be relocated. So if a "C array of doubles" is not native to the CPU, then I don't know what is.

Re: Apache Arrow 3.0

#82

Post by [deleted] »

waynesonfire wrote: post #63
TuringTest wrote: post #50

Earlier quoted context omitted.

If it works as a universal intermediate exchange language, it could help standardize connections among disparate systems. When you have N systems, it takes N^2 translators to build direct connections to transfer data between them; but it only takes N translators if all them can talk the same exchange language.

can you define what at translator is? I don't understand the complexity you're constructing. I have N systems and they talk protobuf. What's the problem?

[deleted]

Re: Apache Arrow 3.0

#83

Post by mumblemumble »

waynesonfire wrote: post #65
mumblemumble wrote: post #61

Earlier quoted context omitted.

It's not just a serde. One of its key use cases is eliminating serde.

I just don't believe you. My CPU doesn't understand Apache Arrow 3.0.

So, there are several components to Arrow. One of them transfers data using IPC, and naturally needs to serialize. The other uses shared memory, which eliminates the need for serde.

Sadly, the latter isn't (yet) well supported anywhere but Python and C++. If you can/do use it, though, data are just kept as as arrays in memory. Which is exactly what the CPU wants to see.

Re: Apache Arrow 3.0

#84

Post by BadInformatics »

liminal wrote: post #64

Would really love to see first class support for Javascript/Typescript for data visualization purposes. The columnar format would naturally lend itself to an Entity-Component style architecture with TypedArrays.

https://arrow.apache.org/docs/js/ has existed for a while and uses typed arrays under the hood. It's a bit of a chunky dependency, but if you're at the point where that level of throughput is required bundle size is probably not a big deal.

Re: Apache Arrow 3.0

#85

Post by breck »

georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Arrow is definitely one of the top 10 new things I'm most excited about in the data science space, but not sure I'd call it the most important thing. ;)

It is pretty awesome, however, particularly for folks like me that are often hopping between Python/R/Javascript. I've definitely got in on the roadmap for all my data science libraries.

Btw, Arquero from that UW lab looks really neat as well, and is supporting Arrow out of the gate (https://github.com/uwdata/arquero).

Re: Apache Arrow 3.0

#86

Post by throwaway894345 »

wesm wrote: post #81
waynesonfire wrote: post #75

Earlier quoted context omitted.

Of course there is. There is always deserialization. The data format is most definitely not native to the CPU.

I challenge you to have a closer look at the project. Deserialization by definition requires bytes or bits to be relocated from their position in the wire protocol to other data structures which are used for processing. Arrow does not require any bytes or bits to be relocated. So if a "C array of doubles" is not native to the CPU, then I don't know what is.

Perhaps "zero-copy" is a more precise or well-defined term?

Re: Apache Arrow 3.0

#87

Post by BadInformatics »

makapuf wrote: post #14
adgjlsfhk1 wrote: post #6

Earlier quoted context omitted.

The big thing is that it is one of the first standardized, cross language binary data formats. CSV is an OK text format, but parsing it is really slow because of string escaping. The files it produces are also pretty big since it's text. Arrow is really fast to parse (up to 1000x faster than CSV), supports data compression, enough data-types to be useful, and deals with metadata well. The closest competitor is probab…

Seems nice. How does it compare to hdf5?

HDF5 is pretty terrible as a wire format, so it's not a 1-1 comparison to Arrow. Generally people are not going to be saving Arrow data to disk either (though you can with the IPC format), but serializing to a more compact representation like Parquet.

Re: Apache Arrow 3.0

#88

Post by staticassertion »

sethhochberg wrote: post #66
rubicon33 wrote: post #54

Earlier quoted context omitted.

Wait so you're telling me I could store data as a PDF file, and access it easily / quickly as SQL?

If you found/wrote a adapter to translate your structured PDF into Arrow's format, yes - the idea is that you can wire up anything that can produce Arrow data to anything that can consume Arrow data.

I'm kinda confused. Is that not the case for literally everything? "You can send me data of format X, all I ask is that you be able to produce format X" ?

I'm assuming that I'm missing something fwiw, not trying to diminish the value.

Re: Apache Arrow 3.0

#89

Post by ianmcook »

snicker7 wrote: post #76
didip wrote: post #58

Earlier quoted context omitted.

Question, doesn't Parquet already do that?

Yes. But parquet is now based on Apache Arrow.

Parquet is not based on Arrow. The Parquet libraries are built into Arrow, but the two projects are separate and Arrow is not a dependency of Parquet.

Re: Apache Arrow 3.0

#90

Post by BadInformatics »

IshKebab wrote: post #78
Lucasoato wrote: post #24

Earlier quoted context omitted.

Do you mean the Parquet format? I don't think Parquet is immature, it is used in so many enterprise environments, it's is one of the few columnar file format for batch analysis and processing. It preforms so well... But I'm curious to know your opinion on this, so feel free to add some context to your position!

Yeah I do. For example Apache Arrow supports in memory compression. But Parquet does not support that. I had to look through the code to find that out, and I found many instances of basically `throw "Not supported"`. And yeah as I said the documentation is just non-existent. If you are already using Arrow, or you absolutely must use a columnar file format then it's probably a good option.

Is that a problem in the Parquet format or in PyArrow? My understanding is that Parquet is primarily meant for on-disk storage (hence the default on-disk compression), so you'd read into Arrow for in-memory compression or IPC.
Post reply on HN