Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow is 10 years old

arrow.apache.org

Post reply on HN
Print view View original

31–40 of 74 posts

Re: Apache Arrow is 10 years old

#31

Post by thinkharderdev »

kccqzy wrote: post #25
mempko wrote: post #18

We use Apache Arrow at my company and it's fantastic. The performance is so good. We have terabytes of time-series financial data and use arrow to store it and process it.

We use Apache Arrow at my company too. It is part of a migration from an old in-house format. When it works it’s good. But there are just way too many bugs in Arrow. For example: a basic arrow computation on strings segfaults because the result does not fit in Arrow’s string type, only the large string type. Instead of casting it or asking the user to cast it, it just segfaults. Another example: a different basic ope…

This will obviously depend on which implementation you use. Using the rust arrow-rs crate you at least get panics when you overflow max buffer sizes. But one of my enduring annoyances with arrow is that they use signed integer types for buffer offsets and the like. I understand why it has to be that way since it's intended to be cross-language and not all languages have unsigned integer types. But it does lead to lots of very weird bugs when you are working in a native language and casting back and forth from signed to unsigned types. I spent a very frustrating day tracking down this one in particular https://github.com/apache/datafusion/issues/15967

Re: Apache Arrow is 10 years old

#32

Post by outside1234 »

aynyc wrote: post #30
tosh wrote: post #27

Earlier quoted context omitted.

parquet is optimized for storage and compresses well (=> smaller files) feather is optimized for fast reading

Given the cost of storage is getting cheaper, wouldn't most firms want to use feather for analytic performance? But everyone uses parquet.

What people have done in the face of cheaper storage is store more data.

Re: Apache Arrow is 10 years old

#33

Post by yencabulator »

aynyc wrote: post #30
tosh wrote: post #27

Earlier quoted context omitted.

parquet is optimized for storage and compresses well (=> smaller files) feather is optimized for fast reading

Given the cost of storage is getting cheaper, wouldn't most firms want to use feather for analytic performance? But everyone uses parquet.

You can, still, gain a lot of performance by doing less I/O.

Re: Apache Arrow is 10 years old

#34

Post by yencabulator »

aynyc wrote: post #26
dionian wrote: post #23

Earlier quoted context omitted.

https://stackoverflow.com/questions/48083405/what-are-the-di...

I read that. But afaik, feather format is stable now. Hence my confusion. I use parquet at work a lot, where we store a lot of time series financial data. We like it. Creating the Parquet data is a pain since it's not append-able.

Generally Parquet files are combined in an LSM style, compacting smaller files into larger ones. Parquet isn't really meant for the "journal" of level-0 append-one-record style storage, it's meant for the levels that follow.

Re: Apache Arrow is 10 years old

#35

Post by shadow28 »

mandeepj wrote: post #19
skeeter2020 wrote: post #7

Earlier quoted context omitted.

>> some performance comparisons vs sqlite. That's not really the purpose; it's really a language-independent format so that you don't need to change it for say, a dataframe or R. It's columnar because for analytics (where you do lots of aggregations and filtering) this is way more performant; the data is intentionally stored so the target columns are continuous. You probably already know, but the analytics equivalent…

> Arrow can also eliminate the need to serialize/de-serialize data when sharing (ex: a high performance data pipeline) because different consumers / tools / operations can use the same memory representation as-is. Not sure if I misunderstood, what are the chances those different consumers / tools / operations are running in your memory space?

Arrow supports zero-copy data sharing - checkout the Arrow IPC format and Arrow Flight.

Re: Apache Arrow is 10 years old

#36

Post by twic »

tosh wrote: post #27
aynyc wrote: post #21

What's the difference between feather and parquet in terms of usage? I get the design philosophy, but how would you use them differently?

parquet is optimized for storage and compresses well (=> smaller files) feather is optimized for fast reading

And now there's Lance! https://lance.org/

Re: Apache Arrow is 10 years old

#38

Post by aynyc »

yencabulator wrote: post #34
aynyc wrote: post #26

Earlier quoted context omitted.

I read that. But afaik, feather format is stable now. Hence my confusion. I use parquet at work a lot, where we store a lot of time series financial data. We like it. Creating the Parquet data is a pain since it's not append-able.

Generally Parquet files are combined in an LSM style, compacting smaller files into larger ones. Parquet isn't really meant for the "journal" of level-0 append-one-record style storage, it's meant for the levels that follow.

So feather for journaling and parquet for long term processing?

Re: Apache Arrow is 10 years old

#40

Post by dionian »

aynyc wrote: post #26
dionian wrote: post #23

Earlier quoted context omitted.

https://stackoverflow.com/questions/48083405/what-are-the-di...

I read that. But afaik, feather format is stable now. Hence my confusion. I use parquet at work a lot, where we store a lot of time series financial data. We like it. Creating the Parquet data is a pain since it's not append-able.

Have you considered something like iceberg tables?
Post reply on HN