What's the difference between feather and parquet in terms of usage? I get the design philosophy, but how would you use them differently?
Apache Arrow is 10 years old
21–30 of 74 posts
Re: Apache Arrow is 10 years old
#22I like arrow for its type system. It's efficient, complete and does not have "infinite precision decimals". Considering Postgres's decimal encoding, using i256 as the backing type is so much saner approach.
Re: Apache Arrow is 10 years old
#23What's the difference between feather and parquet in terms of usage? I get the design philosophy, but how would you use them differently?
Re: Apache Arrow is 10 years old
#24We use Apache Arrow at my company and it's fantastic. The performance is so good. We have terabytes of time-series financial data and use arrow to store it and process it.
stumbled upon it recently while optimizing parquet writes. It worked flawlessly and 10-20x'd my throughput
Re: Apache Arrow is 10 years old
#25We use Apache Arrow at my company and it's fantastic. The performance is so good. We have terabytes of time-series financial data and use arrow to store it and process it.
We use Apache Arrow at my company too. It is part of a migration from an old in-house format. When it works it’s good. But there are just way too many bugs in Arrow. For example: a basic arrow computation on strings segfaults because the result does not fit in Arrow’s string type, only the large string type. Instead of casting it or asking the user to cast it, it just segfaults. Another example: a different basic operation causes an exception complaining about negative buffer sizes when using variable-length binary type.
Re: Apache Arrow is 10 years old
#26What's the difference between feather and parquet in terms of usage? I get the design philosophy, but how would you use them differently?
https://stackoverflow.com/questions/48083405/what-are-the-di...
I read that. But afaik, feather format is stable now. Hence my confusion. I use parquet at work a lot, where we store a lot of time series financial data. We like it. Creating the Parquet data is a pain since it's not append-able.
Re: Apache Arrow is 10 years old
#27What's the difference between feather and parquet in terms of usage? I get the design philosophy, but how would you use them differently?
parquet is optimized for storage and compresses well (=> smaller files)
feather is optimized for fast reading
Re: Apache Arrow is 10 years old
#28Earlier quoted context omitted.
>> some performance comparisons vs sqlite. That's not really the purpose; it's really a language-independent format so that you don't need to change it for say, a dataframe or R. It's columnar because for analytics (where you do lots of aggregations and filtering) this is way more performant; the data is intentionally stored so the target columns are continuous. You probably already know, but the analytics equivalent…
> Arrow can also eliminate the need to serialize/de-serialize data when sharing (ex: a high performance data pipeline) because different consumers / tools / operations can use the same memory representation as-is. Not sure if I misunderstood, what are the chances those different consumers / tools / operations are running in your memory space?
If you are in control of two processes on a single machine instance, you could share the memory between a writer and a read-only consumer.
The key phrase though would seem to be “memory representation”m and not “same memory”. You can spit the in-memory representation out to an Arrow file or an Arrow stream, take it in, and it’s in the same memory layout in the other program. That’s kind of the point of Arrow. It’s a standard memory layout available across applications and even across languages, which can be really convenient.
Re: Apache Arrow is 10 years old
#29[deleted]
Re: Apache Arrow is 10 years old
#30What's the difference between feather and parquet in terms of usage? I get the design philosophy, but how would you use them differently?
parquet is optimized for storage and compresses well (=> smaller files) feather is optimized for fast reading
Given the cost of storage is getting cheaper, wouldn't most firms want to use feather for analytic performance? But everyone uses parquet.