Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Demystifying Apache Arrow (2020)

robinlinacre.com

Post reply on HN
Print view View original

21–30 of 49 posts

Re: Demystifying Apache Arrow (2020)

#21

Post by d_burfoot »

Can someone comment on the code quality of Arrow vs other Apache data engineering tools?

I have been burned so many times by amateur hour software engineering failures from the Apache world, that it’s very hard for me to ever willingly adopt anything from that brand again. Just put it in gripped JSon or TSV and hey, if there’s a performance penalty, it’s better to pay a bit more for cloud compute than hate your job because of some nonsense dependency issue caused by an org.Apache library failing to follow proper versioning guidelines.

Re: Demystifying Apache Arrow (2020)

#22

Post by intelVISA »

tveita wrote: post #14
mjburgess wrote: post #2

> Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds If someone has a code example to this effect, I'd be greatful. I was once engaged in a salesy pitch by a cloud advocate that BigQuery (et al.) can "process a billion rows a second". I tried to create an SQLite example with a billion rows to show that this isn't impressive, but I gave up after some obstacles to gener…

Clickhouse or DuckDB are databases I would look at that support this use case pretty much "out of the box" E.g. https://benchmark.clickhouse.com has some query times for a 100 million row dataset.

Having used all three I'd go with Clickhouse/DuckDB over Arrow every time.

Re: Demystifying Apache Arrow (2020)

#24

Post by RobinL »

rr888 wrote: post #20

I always thought the file format was going to be tightly bound to Arrow but looks like they aren't encouraging feather. Should we just be using Parquet for file storage?

Yes - save to parquet. From the OP:

"Why not just persist the data to disk in Arrow format, and thus have a single, cross-language data format that is the same on-disk and in-memory? One of the biggest reasons is that Parquet generally produces smaller data files, which is more desirable if you are IO-bound. This will especially be the case if you are loading data from cloud storage like such as AWS S3.

Julien LeDem explains this further in a blog post discussing the two formats:

>> The trade-offs for columnar data are different for in-memory. For data on disk, usually IO dominates latency, which can be addressed with aggressive compression, at the cost of CPU. In memory, access is much faster and we want to optimise for CPU throughput by paying attention to cache locality, pipelining, and SIMD instructions. https://www.kdnuggets.com/2017/02/apache-arrow-parquet-colum..."

Re: Demystifying Apache Arrow (2020)

#25

Post by sanderjd »

intelVISA wrote: post #22
tveita wrote: post #14

Earlier quoted context omitted.

Clickhouse or DuckDB are databases I would look at that support this use case pretty much "out of the box" E.g. https://benchmark.clickhouse.com has some query times for a 100 million row dataset.

Having used all three I'd go with Clickhouse/DuckDB over Arrow every time.

Oh interesting - why?

Re: Demystifying Apache Arrow (2020)

#26

Post by sanderjd »

RobinL wrote: post #17

Author here. Since I wrote this, Arrow seems to be be more and more pervasive. As a data engineer, the adoption of Arrow (and parquet) as a data exchange format has so much value. It's amazing how much time me and colleagues have spent on data type issues that have arisen from the wide range of data tooling (R, Pandas, Excel etc. etc.). So much so that I try to stick to parquet, using SQL where possible to easily pre…

Since you know a bunch about this, I'm going to ask you a question that I was about to research: If I have a dataset in memory in Arrow, but I want to cache it to disk to read back in later, what is the most efficient way to do that at this moment in time? Is it to write to parquet and read the parquet back into memory, or is there a more efficient way to write the native Arrow format such that it can be read back in directly? I think this sounds kind of like Flight, except that my understanding is that is intended for moving the data across a network rather than temporally across a disk.

Re: Demystifying Apache Arrow (2020)

#27

Post by sanderjd »

d_burfoot wrote: post #21

Can someone comment on the code quality of Arrow vs other Apache data engineering tools? I have been burned so many times by amateur hour software engineering failures from the Apache world, that it’s very hard for me to ever willingly adopt anything from that brand again. Just put it in gripped JSon or TSV and hey, if there’s a performance penalty, it’s better to pay a bit more for cloud compute than hate your job b…

Arrow (and the ecosystem around it that I've looked into, namely DataFusion) seems really solid and well-engineered to me.

Re: Demystifying Apache Arrow (2020)

#29

Post by RobinL »

sanderjd wrote: post #26
RobinL wrote: post #17

Author here. Since I wrote this, Arrow seems to be be more and more pervasive. As a data engineer, the adoption of Arrow (and parquet) as a data exchange format has so much value. It's amazing how much time me and colleagues have spent on data type issues that have arisen from the wide range of data tooling (R, Pandas, Excel etc. etc.). So much so that I try to stick to parquet, using SQL where possible to easily pre…

Since you know a bunch about this, I'm going to ask you a question that I was about to research: If I have a dataset in memory in Arrow, but I want to cache it to disk to read back in later, what is the most efficient way to do that at this moment in time? Is it to write to parquet and read the parquet back into memory, or is there a more efficient way to write the native Arrow format such that it can be read back in…

I'm not an expert in the nuts and bolts of Arrow, but I think you have two options:

- Save to feather format. Feather format is essentially the same thing as the Arrow in-memory format. This is uncompressed and so if you have super fast IO, it'll read back to memory faster, or at least, with minimal CPU usage.

- Save to compressed parquet format. Because you're often IO bound, not CPU bound, this may read back to memory faster, at the expense of the CPU usage of decompressing.

On a modern machine with a fast SSD, I'm not sure which would be faster. If you're saving to remote blob storage e.g. S3, parquet will almost certainly be faster.

See also https://news.ycombinator.com/item?id=34324649

Re: Demystifying Apache Arrow (2020)

#30

Post by mempko »

RobinL wrote: post #24
rr888 wrote: post #20

I always thought the file format was going to be tightly bound to Arrow but looks like they aren't encouraging feather. Should we just be using Parquet for file storage?

Yes - save to parquet. From the OP: "Why not just persist the data to disk in Arrow format, and thus have a single, cross-language data format that is the same on-disk and in-memory? One of the biggest reasons is that Parquet generally produces smaller data files, which is more desirable if you are IO-bound. This will especially be the case if you are loading data from cloud storage like such as AWS S3. Julien LeDem…

I opted to store feather for one particular reason. You can open it using mmap and randomly index the data without having to load it all in memory. Also the data I have isn't very compressible to begin with, so the cpu cost vs data savings of parquet don't make sense. This only makes sense in that narrow use case.
Post reply on HN