131–140 of 204 posts
Can someone ELI5 what problems are best solved by apache arrow?
Not having to maintain data type definitions when sending around the data, nor caring whether my colleagues were using R or Python, worked great.
At any rate, archive.org managed to grab it https://web.archive.org/web/20210203194945/https://arrow.apa...
Earlier quoted context omitted.
Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...
Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases? * BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the…
* BigQuery: no
* Redshift: no
* Spark SQL: no
* Snowflake: no
* Clickhouse: no
* Dremio: no
* Impala: no
* Presto: no
I have a question about whether it would fit this use-case:
* I need a SUPER fast KV-store.
* I'm on a single machine.
* Keys are 10-bytes if you compress (or strings with 32 characters if you don't), unfortunately I can't store it as an 8-byte int. sqlite said it supports arbitrary precision numerics, but then I got burned finding out that casts integers to arbitrary precision floats and only keeps the first 14 digits of precision :\
* Values are 4-byte ints. Maybe 3 4-byte ints.
* I have maybe 10B - 100B rows.
* I need super fast lookup and depending upon my machine can't always cache this in memory, might need to work from disk.
Would arrow be useful for this? Currently just using sqlite.
Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…
Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...
I got interested in Arrow recently after reading this blog post showing that Arrow (and Ray) are much faster than Pickle: https://rise.cs.berkeley.edu/blog/fast-python-serialization-... I have a question about whether it would fit this use-case: * I need a SUPER fast KV-store. * I'm on a single machine. * Keys are 10-bytes if you compress (or strings with 32 characters if you don't), unfortunately I can't store it as…
That isn't saying much, though.
https://www.benfrederickson.com/images/python-serialization/...
I got interested in Arrow recently after reading this blog post showing that Arrow (and Ray) are much faster than Pickle: https://rise.cs.berkeley.edu/blog/fast-python-serialization-... I have a question about whether it would fit this use-case: * I need a SUPER fast KV-store. * I'm on a single machine. * Keys are 10-bytes if you compress (or strings with 32 characters if you don't), unfortunately I can't store it as…
Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…
Are these execution engines internally using Arrow columnar format or are they just exposing Arrow as a client wire format? AFAIK Spark and Presto does not use Arrow as execution columnar format, but just data sources/sinks.
Earlier quoted context omitted.
I just don't believe you. My CPU doesn't understand Apache Arrow 3.0.
So, there are several components to Arrow. One of them transfers data using IPC, and naturally needs to serialize. The other uses shared memory, which eliminates the need for serde. Sadly, the latter isn't (yet) well supported anywhere but Python and C++. If you can/do use it, though, data are just kept as as arrays in memory. Which is exactly what the CPU wants to see.
Earlier quoted context omitted.
For those wondering what a SerDe is: https://docs.serde.rs/serde/
The term likely predates the Rust implementation. SerDe is Serializer & Deserializer, which could be any framework or tool that allows the serialization and deserialization of data. I first came across the concept in Apache Hive.
