Apache Arrow 3.0 • Hacker News Board

Re: Apache Arrow 3.0

#91

Post by liminal » Wed, Feb 03, 2021, 10:48 PM UTC

BadInformatics wrote: post #84

liminal wrote: post #64

Would really love to see first class support for Javascript/Typescript for data visualization purposes. The columnar format would naturally lend itself to an Entity-Component style architecture with TypedArrays.

https://arrow.apache.org/docs/js/ has existed for a while and uses typed arrays under the hood. It's a bit of a chunky dependency, but if you're at the point where that level of throughput is required bundle size is probably not a big deal.

I was mostly looking at this: https://arrow.apache.org/docs/status.html

Re: Apache Arrow 3.0

#92

Post by kristjansson » Wed, Feb 03, 2021, 10:50 PM UTC

wesm wrote: post #79

kristjansson wrote: post #77

Earlier quoted context omitted.

Not GP post, but it might have been better stated as 'eliminating serde overhead'. Arrow's RPC serialization [1] is basically Protobuf, with a whole lot of hacks to eliminate copies on both ends of the wire. So it's still 'serde', but markedly more efficient for large blocks of tabular-ish data. [1]: https://arrow.apache.org/docs/format/Flight.html

> Arrow's serialization is Protobuf Incorrect. Only Arrow Flight embeds the Arrow wire format in a Protocol Buffer, but the Arrow protocol itself does not use Protobuf.

Apologies, off base there. Edited with a pointer to Flight :)

Re: Apache Arrow 3.0

#93

Post by [deleted] » Wed, Feb 03, 2021, 10:51 PM UTC

staticassertion wrote: post #88

sethhochberg wrote: post #66

Earlier quoted context omitted.

If you found/wrote a adapter to translate your structured PDF into Arrow's format, yes - the idea is that you can wire up anything that can produce Arrow data to anything that can consume Arrow data.

I'm kinda confused. Is that not the case for literally everything? "You can send me data of format X, all I ask is that you be able to produce format X" ? I'm assuming that I'm missing something fwiw, not trying to diminish the value.

[deleted]

Re: Apache Arrow 3.0

#94

Post by waynesonfire » Wed, Feb 03, 2021, 10:52 PM UTC

wesm wrote: post #81

waynesonfire wrote: post #75

Earlier quoted context omitted.

Of course there is. There is always deserialization. The data format is most definitely not native to the CPU.

I challenge you to have a closer look at the project. Deserialization by definition requires bytes or bits to be relocated from their position in the wire protocol to other data structures which are used for processing. Arrow does not require any bytes or bits to be relocated. So if a "C array of doubles" is not native to the CPU, then I don't know what is.

CPUs come in many flavors. One area where they differ is in the way that bytes of a word are represented in memory. Two common formats are Big Endian and Little Endian. This is an example where a "C array of doubles" would be incompatible and some form of deserilaziation would be needed.

My understanding is that an apache arrow library provides an API to manipulate the format in a platform agnostic way. But to claim that it eliminates deserialization is false.

Re: Apache Arrow 3.0

#95

Post by humbleMouse » Wed, Feb 03, 2021, 10:57 PM UTC

I worked at a large company a few years ago on a team implementing this. It’s super cool and works great. Definitely where the future is headed

Re: Apache Arrow 3.0

#96

Post by anonyfox » Wed, Feb 03, 2021, 10:57 PM UTC

So if I understand this correctly from an application developers perspective:

- for OLTP tasks, something row based like sqlite is great. Small to medium amounts of data mixed reading/writing with transactions

- for OLAP tasks, arrow looks great. Big amounts of data, faster querying (datafusion) and more compact data files with parquet.

Basically prevent the operational database from growing too large, offload older data to arrow/parquet. Did I get this correct?

Additionally there seem to be further benefits like sharing arrow/parquet with other consumers.

Sounds convincing, I just have two very specific questions:

- if I load a ~2GB collection of items into arrow and query it with datafusion, how much slower will this perform in comparison to my current rust code that holds a large Vec in memory and „queries“ via iter/filter?

- if I want to move data from sqlite to a more permanent parquet „Archive“ file, is there a better way than recreating the whole file or write additional files, like, appending?

Really curious, could find no hints online so far to get an idea.

Re: Apache Arrow 3.0

#97

Post by BadInformatics » Wed, Feb 03, 2021, 10:59 PM UTC

liminal wrote: post #91

BadInformatics wrote: post #84

Earlier quoted context omitted.

https://arrow.apache.org/docs/js/ has existed for a while and uses typed arrays under the hood. It's a bit of a chunky dependency, but if you're at the point where that level of throughput is required bundle size is probably not a big deal.

I was mostly looking at this: https://arrow.apache.org/docs/status.html

Not sure I follow, that page indicates that JS support is pretty good for all but the more obscure features (e.g. decimals) and doesn't mention data visualization at all? Anyhow, I've successfully used https://github.com/vega/vega-loader-arrow for in-browser plots before, and Observable has a fair few notebooks showing how to use the JS API (e.g. https://observablehq.com/@theneuralbit/introduction-to-apach...)

Re: Apache Arrow 3.0

#98

Post by humbleMouse » Wed, Feb 03, 2021, 10:59 PM UTC

staticassertion wrote: post #88

sethhochberg wrote: post #66

Earlier quoted context omitted.

If you found/wrote a adapter to translate your structured PDF into Arrow's format, yes - the idea is that you can wire up anything that can produce Arrow data to anything that can consume Arrow data.

I'm kinda confused. Is that not the case for literally everything? "You can send me data of format X, all I ask is that you be able to produce format X" ? I'm assuming that I'm missing something fwiw, not trying to diminish the value.

The difference is that arrow’s mapping behind the scenes enables automatic translation to any implemented “plugin” that is on the user’s implementation of arrow. You can extend arrows format to make it automatically map to whatever you want, basically.

And it’s all stored in memory - so much faster access to complex data relationships than anything that exists to my knowledge.

Re: Apache Arrow 3.0

#99

Post by [deleted] » Wed, Feb 03, 2021, 11:00 PM UTC

georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

[deleted]

Re: Apache Arrow 3.0

#100

Post by humbleMouse » Wed, Feb 03, 2021, 11:00 PM UTC

anonyfox wrote: post #96

So if I understand this correctly from an application developers perspective: - for OLTP tasks, something row based like sqlite is great. Small to medium amounts of data mixed reading/writing with transactions - for OLAP tasks, arrow looks great. Big amounts of data, faster querying (datafusion) and more compact data files with parquet. Basically prevent the operational database from growing too large, offload older…

I think the best way to think about it is, you have teams where hadoop queries take 5+ hours to run. You port that same data into arrow, now that same query takes 30 seconds.