Can someone ELI5 what problems are best solved by apache arrow?
[deleted]
Apache Arrow 3.0
41–50 of 204 posts
Re: Apache Arrow 3.0
#42I'm surprised they are still making breaking changes, and they plan to make more (they are already working on a 4.0).
I didn't see any breaking changes in the release notes but I may have missed them. Maybe they don't use SemVer?
Arrow uses SemVer, but the library and the data format are versioned separately: https://arrow.apache.org/docs/format/Versioning.html
Re: Apache Arrow 3.0
#43Can someone ELI5 what problems are best solved by apache arrow?
In Perspective (https://github.com/finos/perspective), we use Apache Arrow as a fast, cross-language/cross-network data encoding that is extremely useful for in-browser data visualization and analytics.
Some benefits:
- super fast read/write compared to CSV & JSON (Perspective and Arrow share an extremely similar column encoding scheme, so we can memcpy Arrow columns into Perspective wholesale instead of reading a dataset iteratively).
- the ability to send Arrow binaries as an ArrayBuffer between a Python server and a WASM client, which guarantees compatibility and removes the overhead of JSON serialization/deserialization.
- because Arrow columns are strictly typed, there's no need to infer data types - this helps with speed and correctness.
- Compared to JSON/CSV, Arrow binaries have a super compact encoding that reduces network transport time.
For us, building on top of Apache Arrow (and using it wherever we can) reduces the friction of passing around data between clients, servers, and runtimes in different languages, and allows larger datasets to be efficiently visualized and analyzed in the browser context.
Re: Apache Arrow 3.0
#44Can someone ELI5 what problems are best solved by apache arrow?
The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…
Not only in between processes, but also in between languages in a single process. In this POC I spun up a Python interpreter in a Go process and pass the Arrow data buffer between processes in constant time. https://github.com/nickpoorman/go-py-arrow-bridge
Re: Apache Arrow 3.0
#45If curious see also
2020 https://news.ycombinator.com/item?id=23965209
2018 (a bit) https://news.ycombinator.com/item?id=17383881
2017 https://news.ycombinator.com/item?id=15335462
2017 https://news.ycombinator.com/item?id=15594542 rediscussed recently https://news.ycombinator.com/item?id=25258626
2016 https://news.ycombinator.com/item?id=11118274
Also: related from a couple weeks ago https://news.ycombinator.com/item?id=25824399
related from a few months ago https://news.ycombinator.com/item?id=24534274
related from 2019 https://news.ycombinator.com/item?id=21826974
Re: Apache Arrow 3.0
#46Earlier quoted context omitted.
protobufs still get encoded and decoded by each client when loaded into memory. arrow is a little bit more like "flatbuffers, but designed for common data-intensive columnar access patterns"
Arrow does actually use flatbuffers for metadata storage.
Re: Apache Arrow 3.0
#47Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…
Uhh.. maybe. It's a serde that's trying to be cross-language / platform.
I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.
Re: Apache Arrow 3.0
#48Earlier quoted context omitted.
This is intended for analytical workloads where you're often doing things that can benefit from vectorization (like SIMD). It's much faster to SUM(X) when all values of X are neatly laid out in-memory. It also has the added benefit of eliminating serialization and deserialization of data between processes - a Python process can now write to memory which is read by a C++ process that's doing windowed aggregations, whi…
> It also has the added benefit of eliminating serialization and deserialization of data between processes Is that accurate? It still has to deserialize from apache arrow format to whatever the cpu understands.
The Arrow Feather format is an on-disk representation of Arrow memory. To read a Feather file, Arrow just copies it byte for byte from disk into memory. Or Arrow can memory-map a Feather file so you can operate on it without reading the whole file into memory.
Re: Apache Arrow 3.0
#49Earlier quoted context omitted.
I hope you don’t mind me asking dumb questions, but how does this differ from the role that say Protocol Buffers fills? To my ears they both facilitate data exchange. Are they comparable in that sense?
Better to compare it to Cap'n Proto instead. Arrow data is already laid out in a usable way. For example, an Arrow column of int64s is an 8-byte aligned memory region of size 8*N bytes (plus a bit vector for nullity), ready for random access or vectorized operations. Protobuf, on the other hand, would encode those values as variable-width integers. This saves a lot of space, which might be better for transfer over a…
Protobuf provides the fixed64 type and when combined with `packed` (the default in proto3, optional in proto2) gives you a linear layout of fixed-size values. You would not get natural alignment from protobuf's wire format if you read it from an arbitrary disk or net buffer; to get alignment you'd need to move or copy the vector. Protobuf's C++ generated code provides RepeatedField that behaves in most respects like std::vector, but in as much as protobuf is partly a wire format and partly a library, users are free to ignore the library and use whatever code is most convenient to their application.
TL;DR variable-width numbers in protobuf are optional.
Re: Apache Arrow 3.0
#50Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…
Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.
If it works as a universal intermediate exchange language, it could help standardize connections among disparate systems.
When you have N systems, it takes N^2 translators to build direct connections to transfer data between them; but it only takes N translators if all them can talk the same exchange language.