Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Demystifying Apache Arrow (2020)

robinlinacre.com

Post reply on HN
Print view View original

41–49 of 49 posts

Re: Demystifying Apache Arrow (2020)

#41

Post by reichardt »

hintymad wrote: post #39
RobinL wrote: post #17

Author here. Since I wrote this, Arrow seems to be be more and more pervasive. As a data engineer, the adoption of Arrow (and parquet) as a data exchange format has so much value. It's amazing how much time me and colleagues have spent on data type issues that have arisen from the wide range of data tooling (R, Pandas, Excel etc. etc.). So much so that I try to stick to parquet, using SQL where possible to easily pre…

Thanks for sharing your insights. Any comments on Feather vs Parquet? If we don't need to support tools that can only interact with Parquet, how will Feather pan out as a Parquet alternative (or Feather can't be such alternative at all)?

I recently looked into this as well. Specifically how the two formats differ. As it stands right now the “Feather” file format seems to be a synonym for the Arrow IPC file format or “Arrow files” [0]. There should be basically no overhead while reading into the arrow memory format [1]. Parquet files on the other hand are stored in a different format and therefore occur some overhead while reading into memory but offer more advanced mechanism for on disk encoding and compression [1].

As far as I can tell the main trade-off seems to be around deserialization overhead vs on disk file size. If anyone has more information or experience with the topic I'd love to hear!

[0] https://arrow.apache.org/faq/#what-about-the-feather-file-fo... [1] https://arrow.apache.org/faq/#what-is-the-difference-between...

EDIT:

More information: https://news.ycombinator.com/item?id=34324649

Re: Demystifying Apache Arrow (2020)

#42

Post by RobinL »

reichardt wrote: post #41
hintymad wrote: post #39

Earlier quoted context omitted.

Thanks for sharing your insights. Any comments on Feather vs Parquet? If we don't need to support tools that can only interact with Parquet, how will Feather pan out as a Parquet alternative (or Feather can't be such alternative at all)?

I recently looked into this as well. Specifically how the two formats differ. As it stands right now the “Feather” file format seems to be a synonym for the Arrow IPC file format or “Arrow files” [0]. There should be basically no overhead while reading into the arrow memory format [1]. Parquet files on the other hand are stored in a different format and therefore occur some overhead while reading into memory but offe…

This is also my understanding - see https://news.ycombinator.com/item?id=34324649

Re: Demystifying Apache Arrow (2020)

#43

Post by reichardt »

RobinL wrote: post #42
reichardt wrote: post #41

Earlier quoted context omitted.

I recently looked into this as well. Specifically how the two formats differ. As it stands right now the “Feather” file format seems to be a synonym for the Arrow IPC file format or “Arrow files” [0]. There should be basically no overhead while reading into the arrow memory format [1]. Parquet files on the other hand are stored in a different format and therefore occur some overhead while reading into memory but offe…

This is also my understanding - see https://news.ycombinator.com/item?id=34324649

Thanks! Just stumbled across your comment as well.

Re: Demystifying Apache Arrow (2020)

#46

Post by _frkl »

mempko wrote: post #30
RobinL wrote: post #24

Earlier quoted context omitted.

Yes - save to parquet. From the OP: "Why not just persist the data to disk in Arrow format, and thus have a single, cross-language data format that is the same on-disk and in-memory? One of the biggest reasons is that Parquet generally produces smaller data files, which is more desirable if you are IO-bound. This will especially be the case if you are loading data from cloud storage like such as AWS S3. Julien LeDem…

I opted to store feather for one particular reason. You can open it using mmap and randomly index the data without having to load it all in memory. Also the data I have isn't very compressible to begin with, so the cpu cost vs data savings of parquet don't make sense. This only makes sense in that narrow use case.

I'm doing the same. It's also quite nice for de-duplication, a lot of operations on our data happen on a column basis, and we need to assemble tables that are basically the same, except for one or two computed columns. I usually store all columns in a separate file, and assemble tables on the fly, also memory-mapped. Quite happy with being able to do that. Not sure how easy that would be with parquet.

Re: Demystifying Apache Arrow (2020)

#47

Post by hermitcrab »

I have written a desktop data wrangling/ETL tool for Windows/Mac (Easy Data Transform). It is designed to handle millions of rows (but not billions). Currently it mostly inputs and outputs CSV, Excel, XML and JSON. I am looking to add some additional formats in future, such as SQLite, Parquet or DuckBD. Maybe I need to look at Feather as well? I could also use one of these formats to store intermediate datasets to disk, rather than holding everything in memory. If anyone has any experience in integrating any of these formats into a C++ application on Windows and/or Mac, I would be interested to hear how you got on and what libraries you used.

Re: Demystifying Apache Arrow (2020)

#48

Post by RobinL »

alamb wrote: post #40

Here is another blog post that offers some perspective on the growth of Arrow over the intervening years and future directions: https://www.datawill.io/posts/apache-arrow-2022-reflection/

That's really good, thanks. Better than my blog, actually - the author has a much deeper understanding and I learned a lot by reading it. I was coming at it from the perspective of someone very confused by Arrow, and wrote the blog to help myself understand it!

Re: Demystifying Apache Arrow (2020)

#49

Post by Infernal »

mempko wrote: post #30
RobinL wrote: post #24

Earlier quoted context omitted.

Yes - save to parquet. From the OP: "Why not just persist the data to disk in Arrow format, and thus have a single, cross-language data format that is the same on-disk and in-memory? One of the biggest reasons is that Parquet generally produces smaller data files, which is more desirable if you are IO-bound. This will especially be the case if you are loading data from cloud storage like such as AWS S3. Julien LeDem…

I opted to store feather for one particular reason. You can open it using mmap and randomly index the data without having to load it all in memory. Also the data I have isn't very compressible to begin with, so the cpu cost vs data savings of parquet don't make sense. This only makes sense in that narrow use case.

As someone new to Arrow/columnar DB's, do you mind sharing what kind of data makes sense to use Arrow for, but isn't very compressible?
Post reply on HN