Apache Arrow 3.0

Post by polskibus » Thu, Feb 04, 2021, 7:32 AM UTC

wesm wrote: post #16

georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Microsoft is also on top of this with their Magpie project http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf "A common, efficient serialized and wire format across data engines is a transformational development. Many previous systems and approaches (e.g., [26, 36, 38, 51]) have observed the prohibitive cost of data conversion and transfer, precluding optimizers from exploiting inter-DBMS performance advantages.…

Post by phillc73 » Thu, Feb 04, 2021, 9:03 AM UTC

hodgesrm wrote: post #170

nomel wrote: post #111

Earlier quoted context omitted.

Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...

What are the use cases where multi-dimensional arrays are important and why?

Post by zX41ZdbW » Thu, Feb 04, 2021, 9:54 AM UTC

wesm wrote: post #130

nomel wrote: post #111

Earlier quoted context omitted.

Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...

Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases? * BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the…

Post by evanpw » Thu, Feb 04, 2021, 10:54 AM UTC

bobbylarrybobby wrote: post #155

MrPowers wrote: post #10

Earlier quoted context omitted.

Exactly. Some specific examples. Read a Parquet file into a Pandas DataFrame. Then read the Pandas DataFrame into a Spark DataFrame. Spark & Pandas are using the same Arrow memory format, so no serde is needed. See the "Standardization Saves" diagram here: https://arrow.apache.org/overview/

Is this merely planned, or does pandas now use Arrow’s format? I was under the impression that pandas was mostly numpy under the hood with some tweaks to handle some of the newer functionality like nullable arrays. But you’re saying that arrow data can be used by pandas without conversion or copying into new memory?

Post by prepend » Thu, Feb 04, 2021, 11:59 AM UTC

NewJazz wrote: post #169

prepend wrote: post #11

Earlier quoted context omitted.

I recently found it useful for the dumbest reason. A dataset was about 3GB as a CSV and 20MB as a parquet file created and consumed by arrow. The file also worked flawlessly across different environments and languages. So it’s a good transport tool. It also happens to be fast to load and query, but I only used it because of the compact way it stores data without any hoops to jump through. Of course one might say that…

Parquet is not Arrow. Parquet has optimizations for storage size at the expense of compute readiness. Arrow maximizes computation efficiency, at the expense of storage size. Read more here: https://stackoverflow.com/questions/56472727/difference-betw...