Apache Arrow 3.0

Post by est » Thu, Feb 04, 2021, 3:11 AM UTC

wesm wrote: post #130

nomel wrote: post #111

Earlier quoted context omitted.

Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...

Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases? * BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the…

Post by georgewfraser » Thu, Feb 04, 2021, 3:17 AM UTC

breck wrote: post #85

georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Arrow is definitely one of the top 10 new things I'm most excited about in the data science space, but not sure I'd call it the most important thing. ;) It is pretty awesome, however, particularly for folks like me that are often hopping between Python/R/Javascript. I've definitely got in on the roadmap for all my data science libraries. Btw, Arquero from that UW lab looks really neat as well, and is supporting Arrow…

Post by int_19h » Thu, Feb 04, 2021, 3:33 AM UTC

est wrote: post #157

mumblemumble wrote: post #61

Earlier quoted context omitted.

It's not just a serde. One of its key use cases is eliminating serde.

Arrow had it selling points as non-serde. But I am wondering how does it achieve no-serde with Python? By allocating PyObject cleverly with a network packet buffer? If I convert an Arrow int8 array to normal python list of int's, will this involve copying?

Post by alisonkisk » Thu, Feb 04, 2021, 3:45 AM UTC

cornstalks wrote: post #57

rcoveson wrote: post #25

Earlier quoted context omitted.

Better to compare it to Cap'n Proto instead. Arrow data is already laid out in a usable way. For example, an Arrow column of int64s is an 8-byte aligned memory region of size 8*N bytes (plus a bit vector for nullity), ready for random access or vectorized operations. Protobuf, on the other hand, would encode those values as variable-width integers. This saves a lot of space, which might be better for transfer over a…

> Think of Arrow as standardized shared memory using struct-of-arrays layout, Cap'n Proto as standardized shared memory using array-of-structs layout I just want to say thank you for this part of the sentence. I understand struct-of-arrays vs array-of-structs, and now I finally understand what the heck Arrow is.

Post by NewJazz » Thu, Feb 04, 2021, 4:10 AM UTC

prepend wrote: post #11

mushufasa wrote: post #3

Can someone ELI5 what problems are best solved by apache arrow?

I recently found it useful for the dumbest reason. A dataset was about 3GB as a CSV and 20MB as a parquet file created and consumed by arrow. The file also worked flawlessly across different environments and languages. So it’s a good transport tool. It also happens to be fast to load and query, but I only used it because of the compact way it stores data without any hoops to jump through. Of course one might say that…