Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow 3.0

arrow.apache.org

Post reply on HN
Print view View original

141–150 of 204 posts

Re: Apache Arrow 3.0

#141

Post by jzer0cool »

neovintage wrote: post #7
mushufasa wrote: post #3

Can someone ELI5 what problems are best solved by apache arrow?

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…

I'm curious too. Does this mean data is first normalized into this "columnar format" as the primary source and all applications are purely working off this format?

I do see yet clearly how the data is being transferred if no serializing/deserializing is taking place if someone here can help fill in further. It almost sounds like there is some specialized bridge for the data transfer and I don't have the right words for it.

Re: Apache Arrow 3.0

#142

Post by rscho »

mumblemumble wrote: post #119
chenster wrote: post #116

Why no love for PHP I wonder? Don't see a supported library there.

If you're successfully doing data science or data engineering in PHP, you're already a god among informaticians, and don't need any extra help.

Hiya, a bit of OT: I saw your comment about type systems in data science the other day (https://news.ycombinator.com/item?id=25923839). From what I understood, it seems you want a contract system, wouldn't you think? The reason I'm asking is that I'm fishing for opinions on building data science infra in Racket (and saw your deleted comment in https://news.ycombinator.com/item?id=26008869 so thought you'd perhaps be interested), and Racket (and R) dataframes happen to support contracts on their columns.

Re: Apache Arrow 3.0

#143

Post by jzer0cool »

tyingq wrote: post #136
bravura wrote: post #134

I got interested in Arrow recently after reading this blog post showing that Arrow (and Ray) are much faster than Pickle: https://rise.cs.berkeley.edu/blog/fast-python-serialization-... I have a question about whether it would fit this use-case: * I need a SUPER fast KV-store. * I'm on a single machine. * Keys are 10-bytes if you compress (or strings with 32 characters if you don't), unfortunately I can't store it as…

"much faster than Pickle" That isn't saying much, though. https://www.benfrederickson.com/images/python-serialization/...

Unrelated: What is this *pickle* term origin?

Python errors sometimes generate some "Could not pickle" errors and not sure what it tries to convey ...

Re: Apache Arrow 3.0

#145

Post by tyingq »

jzer0cool wrote: post #143
tyingq wrote: post #136

Earlier quoted context omitted.

"much faster than Pickle" That isn't saying much, though. https://www.benfrederickson.com/images/python-serialization/...

Unrelated: What is this *pickle* term origin? Python errors sometimes generate some "Could not pickle" errors and not sure what it tries to convey ...

Pickling a cucumber stores and preserves it for later use, so the term is used for serialization. Still an odd analogy though, since you can't unpickle a pickle. I've heard freeze/thaw which is at least a little better.

Re: Apache Arrow 3.0

#146

Post by jzer0cool »

georgewfraser wrote: post #60
jayd16 wrote: post #30

Can someone dig into the pros and cons of the columnar aspect of Arrow? To some degree there are many other data transfer formats but this one seems to promote its columnar orientation. Things like eg. protobuffers support hierarchical data which seems like a superset of columns. Is there a benefit to a column based format? Is it an enforced simplification to ensure greater compatibility or is there some other reason…

The main benefit of a columnar representation in memory is it's more cache friendly for a typical analytical workload. For example, if I have a dataframe: (A int, B int, C int, D int) And I write: A + B In a columnar representation, all the As are next to each other, and all the Bs are next to each other, so the process of (A and B in memory) => (A and B in CPU registers) => (addition) => (A + B result back to memory…

Could you expand on this more columnar data and row data? I missed something here how the data is organized and what you mean by the C,D values getting dragged along.

Re: Apache Arrow 3.0

#147

Post by staticautomatic »

humbleMouse wrote: post #98
staticassertion wrote: post #88

Earlier quoted context omitted.

I'm kinda confused. Is that not the case for literally everything? "You can send me data of format X, all I ask is that you be able to produce format X" ? I'm assuming that I'm missing something fwiw, not trying to diminish the value.

The difference is that arrow’s mapping behind the scenes enables automatic translation to any implemented “plugin” that is on the user’s implementation of arrow. You can extend arrows format to make it automatically map to whatever you want, basically. And it’s all stored in memory - so much faster access to complex data relationships than anything that exists to my knowledge.

Could I write a plug-in that mapped to Cypher? I’ve got a graph use case in mind where I want to use RedisGraph but don’t feel comfy with Redis as a primary DB and would totally consider a columnar store as a primary if I didn’t have to serialize.

Re: Apache Arrow 3.0

#149

Post by mumblemumble »

superdimwit wrote: post #139
mumblemumble wrote: post #83

Earlier quoted context omitted.

So, there are several components to Arrow. One of them transfers data using IPC, and naturally needs to serialize. The other uses shared memory, which eliminates the need for serde. Sadly, the latter isn't (yet) well supported anywhere but Python and C++. If you can/do use it, though, data are just kept as as arrays in memory. Which is exactly what the CPU wants to see.

Shared memory format is supported in Julia too!

Oh, that's fantastic to hear. Right now I'm living in Python because that's the galactic center, but I've also been anxious to find a low-cost escape hatch that doesn't just lead to C++.

Re: Apache Arrow 3.0

#150

Post by mumblemumble »

dlnovell wrote: post #114
waynesonfire wrote: post #94

Earlier quoted context omitted.

CPUs come in many flavors. One area where they differ is in the way that bytes of a word are represented in memory. Two common formats are Big Endian and Little Endian. This is an example where a "C array of doubles" would be incompatible and some form of deserilaziation would be needed. My understanding is that an apache arrow library provides an API to manipulate the format in a platform agnostic way. But to claim…

I wish I had your confidence, to argue with Wes McKinney about the details of how Arrow works

Iunno. That looks like "where angels fear to tread" territory to me.
Post reply on HN