Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow 3.0

arrow.apache.org

Post reply on HN
Print view View original

171–180 of 204 posts

Re: Apache Arrow 3.0

#171

Post by kylebarron »

int_19h wrote: post #165
est wrote: post #157

Earlier quoted context omitted.

Arrow had it selling points as non-serde. But I am wondering how does it achieve no-serde with Python? By allocating PyObject cleverly with a network packet buffer? If I convert an Arrow int8 array to normal python list of int's, will this involve copying?

I think the idea is that you'd have a Python object that behaves exactly like a list of ints would, but with Arrow as its backing store.

This is similar to Numpy. You operate on Python objects that describe the Numpy structure, but the actual memory is stored in C objects, not Python objects.

Re: Apache Arrow 3.0

#172

Post by alisonkisk »

jzer0cool wrote: post #141
neovintage wrote: post #7

Earlier quoted context omitted.

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…

I'm curious too. Does this mean data is first normalized into this "columnar format" as the primary source and all applications are purely working off this format? I do see yet clearly how the data is being transferred if no serializing/deserializing is taking place if someone here can help fill in further. It almost sounds like there is some specialized bridge for the data transfer and I don't have the right words f…

I think you've got it. Data is shared by passing a pointer to it, so the data doesn't need to be copied to different spots in memory (or if it is it's an efficient block copy not millions of tiny copies).

Re: Apache Arrow 3.0

#173

Post by alisonkisk »

mushufasa wrote: post #121
neovintage wrote: post #7

Earlier quoted context omitted.

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…

is there a strong use case around passing data from a backend to a frontend, e.g. from pandas data frame on the server into a js implementation on the client side, to use in a UI? As opposed to data pipelining among processing servers.

Not as much because you are still paying the overhead of doubt shipping across the internet and through the browser.

Re: Apache Arrow 3.0

#174

Post by alisonkisk »

prepend wrote: post #154
kristianp wrote: post #124

Earlier quoted context omitted.

Did you have to reformat the data to get that size saving?

Not at all. It was just swapping out readcsv with readparquet in R and Python. It was painless. Granted my dataset was mostly categorical so that’s why it compressed down so much, but it was a real lifesaver. It was also nice to be able to read while bundles of parquet a into a single dataframe easily. So is nice for “sharding” really big parquets over multiple files. Or fitting under file size limits on git repos.

gzip probably would have been fine for that case even without parquet?

Re: Apache Arrow 3.0

#175

Post by oscardssmith »

mumblemumble wrote: post #149
superdimwit wrote: post #139

Earlier quoted context omitted.

Shared memory format is supported in Julia too!

Oh, that's fantastic to hear. Right now I'm living in Python because that's the galactic center, but I've also been anxious to find a low-cost escape hatch that doesn't just lead to C++.

You should definitely check out Julia then. There are a few parts of the language that use C/C++ libraries (blas and mpfr are the main ones), but 95% of the time, your stack will be Julia all the way down.

Re: Apache Arrow 3.0

#176

Post by ianmcook »

ryanianian wrote: post #132

This link is a 404. Perhaps they weren't intending this post to be public yet? At any rate, archive.org managed to grab it https://web.archive.org/web/20210203194945/https://arrow.apa...

Thanks for the heads up. The post is intended to be up but there's an intermittent error happening. It's been reported to the Apache infrastructure team.

Re: Apache Arrow 3.0

#177

Post by asperous »

cdash wrote: post #112
poorman wrote: post #44

Earlier quoted context omitted.

Not only in between processes, but also in between languages in a single process. In this POC I spun up a Python interpreter in a Go process and pass the Arrow data buffer between processes in constant time. https://github.com/nickpoorman/go-py-arrow-bridge

How is this any better than something like flatbuffers though?

This is explained in the FAQS for Arrow, saying they use flatbuffers internally. Arrow supports more complex data types

Re: Apache Arrow 3.0

#178

Post by [deleted] »

jedbrown wrote: post #59
BenoitP wrote: post #29

Earlier quoted context omitted.

> most of the compute time spent is in serializing and deserializing data. This is to be viewed in light how hardware evolves now. CPU compute power is no longer growing as much (at least for individual cores). But one thing that's still doubling on a regular basis is memory capacity of all kinds (RAM, SSD, etc) and bandwidth of all kinds (PCIe lanes, networking, etc). This divide is getting large and will only conti…

This is backward -- this sort of serialization is overwhelmingly bottlenecked on bandwidth (not CPU). (Multi-core) compute improvements have been outpacing bandwidth improvements for decades and have not stopped. Serialization is a bottleneck because compute is fast/cheap and bandwidth is precious. This is also reflected in the relative energy to move bytes being increasingly larger than the energy to do some arithme…

[deleted]

Re: Apache Arrow 3.0

#179

Post by repsilat »

jzer0cool wrote: post #146
georgewfraser wrote: post #60

Earlier quoted context omitted.

The main benefit of a columnar representation in memory is it's more cache friendly for a typical analytical workload. For example, if I have a dataframe: (A int, B int, C int, D int) And I write: A + B In a columnar representation, all the As are next to each other, and all the Bs are next to each other, so the process of (A and B in memory) => (A and B in CPU registers) => (addition) => (A + B result back to memory…

Could you expand on this more columnar data and row data? I missed something here how the data is organized and what you mean by the C,D values getting dragged along.

To rephrase the sibling comment, if you had an array of four {ABCD} structs, there's basically two ways of storing them on disk:

1. AAAA BBBB CCCC DDDD

2. ABCD ABCD ABCD ABCD

One major heuristic in how CPUs make your code fast is to assume that if you access some memory, you're probably interested in the memory nearby. So when you access the first "A" bit of memory (common to both sequences above), depending on the memory layout you use, the CPU might also be smart and load the next bits into memory too -- maybe the next "AA", maybe "BC".

Depending on your workload, one or the other of those might be faster. If you're only interested in the first ABCD element because you're doing

  SELECT * FROM users WHERE id=$1
then you'll likely want "row-oriented" data -- the #2 scheme above. But if you're interested in all of the A values and none of the values from B/C/D because you're doing

  SELECT AVG(age) FROM users
then you'll likely want something "column-oriented" -- the #1 scheme above.

Re: Apache Arrow 3.0

#180

Post by tomnipotent »

asperous wrote: post #177
cdash wrote: post #112

Earlier quoted context omitted.

How is this any better than something like flatbuffers though?

This is explained in the FAQS for Arrow, saying they use flatbuffers internally. Arrow supports more complex data types

> saying they use flatbuffers internally

Only for IPC support - Arrow data format does not use flatbuffers.

Post reply on HN