Related - recent Apache Arrow support in Julia announcement - https://julialang.org/blog/2021/01/arrow/
Apache Arrow 3.0
151–160 of 204 posts
Re: Apache Arrow 3.0
#152Earlier quoted context omitted.
Unrelated: What is this *pickle* term origin? Python errors sometimes generate some "Could not pickle" errors and not sure what it tries to convey ...
Pickling a cucumber stores and preserves it for later use, so the term is used for serialization. Still an odd analogy though, since you can't unpickle a pickle. I've heard freeze/thaw which is at least a little better.
It's Python. So it's Monty Python.
Re: Apache Arrow 3.0
#153Earlier quoted context omitted.
I wish I had your confidence, to argue with Wes McKinney about the details of how Arrow works
Iunno. That looks like "where angels fear to tread" territory to me.
Hiya, a bit of OT (again, last one promise!): I saw your comment about type systems in data science the other day (https://news.ycombinator.com/item?id=25923839). From what I understood, it seems you want a contract system, wouldn't you think? The reason I'm asking is that I'm fishing for opinions on building data science infra in Racket (and saw your deleted comment in https://news.ycombinator.com/item?id=26008869 so thought you'd perhaps be interested), and Racket (and R) dataframes happen to support contracts on their columns.
Re: Apache Arrow 3.0
#154Earlier quoted context omitted.
I recently found it useful for the dumbest reason. A dataset was about 3GB as a CSV and 20MB as a parquet file created and consumed by arrow. The file also worked flawlessly across different environments and languages. So it’s a good transport tool. It also happens to be fast to load and query, but I only used it because of the compact way it stores data without any hoops to jump through. Of course one might say that…
Did you have to reformat the data to get that size saving?
Not at all. It was just swapping out readcsv with readparquet in R and Python. It was painless. Granted my dataset was mostly categorical so that’s why it compressed down so much, but it was a real lifesaver.
It was also nice to be able to read while bundles of parquet a into a single dataframe easily. So is nice for “sharding” really big parquets over multiple files. Or fitting under file size limits on git repos.
Re: Apache Arrow 3.0
#155Earlier quoted context omitted.
The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…
Exactly. Some specific examples. Read a Parquet file into a Pandas DataFrame. Then read the Pandas DataFrame into a Spark DataFrame. Spark & Pandas are using the same Arrow memory format, so no serde is needed. See the "Standardization Saves" diagram here: https://arrow.apache.org/overview/
Is this merely planned, or does pandas now use Arrow’s format? I was under the impression that pandas was mostly numpy under the hood with some tweaks to handle some of the newer functionality like nullable arrays. But you’re saying that arrow data can be used by pandas without conversion or copying into new memory?
Re: Apache Arrow 3.0
#156Earlier quoted context omitted.
"much faster than Pickle" That isn't saying much, though. https://www.benfrederickson.com/images/python-serialization/...
Unrelated: What is this *pickle* term origin? Python errors sometimes generate some "Could not pickle" errors and not sure what it tries to convey ...
Pickle is the Python module that (de)serializes Python objects to some sort of binary format. Failure to pickle means the object couldn’t be serialized — the most common instance is probably trying to pass a lambda to a multiprocessing worker, because they don’t have names (you should use a def or functions.partial instead).
Re: Apache Arrow 3.0
#157Earlier quoted context omitted.
Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.
It's not just a serde. One of its key use cases is eliminating serde.
Arrow had it selling points as non-serde. But I am wondering how does it achieve no-serde with Python? By allocating PyObject cleverly with a network packet buffer?
If I convert an Arrow int8 array to normal python list of int's, will this involve copying?
Re: Apache Arrow 3.0
#158Earlier quoted context omitted.
HDF5 is pretty terrible as a wire format, so it's not a 1-1 comparison to Arrow. Generally people are not going to be saving Arrow data to disk either (though you can with the IPC format), but serializing to a more compact representation like Parquet.
As I understand, arrow is particularly interesting since it’s wire format can be immediately queried/operated on without deserialization. Would saving an Arrow-structure as parquet not defeat that purpose, since your would need the costly deserialization step again on read? Honest question
The FAQ [1] and this SO answer [2] explain it better than I can, but basically yes. However, the (de)serialization overhead is probably better than most alternative formats you could save to.
[1] https://arrow.apache.org/faq/ [2] https://stackoverflow.com/questions/56472727/difference-betw...
Re: Apache Arrow 3.0
#159Earlier quoted context omitted.
Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...
Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases? * BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the…
I think that implementing good ndim=2 support would already be a huge leap forward, it doesn't have to be something super generic. E.g., given that most of the classic machine learning is essentially using 2-dimensional data (samples x features) as inputs, this is a very common use case.
E.g., as of right now, having to concatenate hundreds of columns manually just in order to pass them to some ml library in a contiguous format is always a pain and often doubles the max ram requirement.
Re: Apache Arrow 3.0
#160Earlier quoted context omitted.
Parquet is not based on Arrow. The Parquet libraries are built into Arrow, but the two projects are separate and Arrow is not a dependency of Parquet.
Arrow has definitely influenced the design of Parquet, they’re meant to compliment each other.
[deleted]