Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow 3.0

arrow.apache.org

Post reply on HN
Print view View original

121–130 of 204 posts

Re: Apache Arrow 3.0

#121

Post by mushufasa »

neovintage wrote: post #7
mushufasa wrote: post #3

Can someone ELI5 what problems are best solved by apache arrow?

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…

is there a strong use case around passing data from a backend to a frontend, e.g. from pandas data frame on the server into a js implementation on the client side, to use in a UI? As opposed to data pipelining among processing servers.

Re: Apache Arrow 3.0

#122

Post by mrfox321 »

cdash wrote: post #112
poorman wrote: post #44

Earlier quoted context omitted.

Not only in between processes, but also in between languages in a single process. In this POC I spun up a Python interpreter in a Go process and pass the Arrow data buffer between processes in constant time. https://github.com/nickpoorman/go-py-arrow-bridge

How is this any better than something like flatbuffers though?

For one, Arrow is columnar.

The idea of zero-copy serialization is shared between Arrow and FlatBuffers.

Re: Apache Arrow 3.0

#123

Post by macksd »

waynesonfire wrote: post #65
mumblemumble wrote: post #61

Earlier quoted context omitted.

It's not just a serde. One of its key use cases is eliminating serde.

I just don't believe you. My CPU doesn't understand Apache Arrow 3.0.

In the best case, your CPU needn't really be involved beyond a call to set up mapped memory. It is (among other things) a common memory format, so you can be working with it in one framework, and then start working with it in another, and it's location and format in memory hasn't changed at all.

Re: Apache Arrow 3.0

#124

Post by kristianp »

prepend wrote: post #11
mushufasa wrote: post #3

Can someone ELI5 what problems are best solved by apache arrow?

I recently found it useful for the dumbest reason. A dataset was about 3GB as a CSV and 20MB as a parquet file created and consumed by arrow. The file also worked flawlessly across different environments and languages. So it’s a good transport tool. It also happens to be fast to load and query, but I only used it because of the compact way it stores data without any hoops to jump through. Of course one might say that…

Did you have to reformat the data to get that size saving?

Re: Apache Arrow 3.0

#125

Post by [deleted] »

chrisweekly wrote: post #68
waynesonfire wrote: post #47

Earlier quoted context omitted.

Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.

For those wondering what a SerDe is: https://docs.serde.rs/serde/

[deleted]

Re: Apache Arrow 3.0

#126

Post by lmeyerov »

nomel wrote: post #111
georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...

FWIW, you can support tensors etc. natively today by using Arrow's structs, such as `column i_am_packed: {x: int32, y: int32, z: int32, time: datetime64[ns], temp: int32}`. This will be a dense, packed representation for whatever fixed type the column has, including dynamically generated. It ends up being quite nice to have that passed around, esp. vs say protobufs.

You wouldn't be sending mixed-length values, and ~all metrics systems I've worked with would be fine there. The nullables end up a win for those. However, you'd need to do a manual (likely zero-copy) casts between the struct-type to whatever tensor-type you're using. Massively popular ml systems like huggingface do this fine afaict for their Arrow-based tensor work. Likewise, as we do a lot of GPU stuff, what's additionally common is compacting the in-memory stuff as big memory blocks ('long recordbatches') instead of CPU-land's typically more fragmented ones, and that ends up making casts even easier. Annoying to have to add an explicit cast for some interop cases, but preserves end-to-end type safety & hasn't been a deal breaker for us. Having to write the cast being annoying/difficult, esp. for whoever does it first.

More frustrating for us has been sparse data and compression controls, but most formats are even worse here..

Re: Apache Arrow 3.0

#127

Post by TheGuyWhoCodes »

kats wrote: post #101

I'm done with Hacker News, you guys just upvote marketing and politics.

I genuinely would like to know what's the issue with a post about a major release of an innovative project. What's political about it? It's not like it's a new release of an enterprise/paid product based on an open source project.

Re: Apache Arrow 3.0

#128

Post by gbrits »

BadInformatics wrote: post #87
makapuf wrote: post #14

Earlier quoted context omitted.

Seems nice. How does it compare to hdf5?

HDF5 is pretty terrible as a wire format, so it's not a 1-1 comparison to Arrow. Generally people are not going to be saving Arrow data to disk either (though you can with the IPC format), but serializing to a more compact representation like Parquet.

As I understand, arrow is particularly interesting since it’s wire format can be immediately queried/operated on without deserialization. Would saving an Arrow-structure as parquet not defeat that purpose, since your would need the costly deserialization step again on read? Honest question

Re: Apache Arrow 3.0

#129

Post by mr_toad »

ianmcook wrote: post #89
snicker7 wrote: post #76

Earlier quoted context omitted.

Yes. But parquet is now based on Apache Arrow.

Parquet is not based on Arrow. The Parquet libraries are built into Arrow, but the two projects are separate and Arrow is not a dependency of Parquet.

Arrow has definitely influenced the design of Parquet, they’re meant to compliment each other.

Re: Apache Arrow 3.0

#130

Post by wesm »

nomel wrote: post #111
georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...

Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases?

* BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues

We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the wild.

Post reply on HN