Apache Arrow is 10 years old • Hacker News Board

Re: Apache Arrow is 10 years old

#71

Post by aynyc » Fri, Feb 13, 2026, 1:48 PM UTC

dionian wrote: post #58

aynyc wrote: post #49

Earlier quoted context omitted.

Yes, but parquet hates small files.

You can't compact? i.e. iceberg maintenance

We might be doing something wrong, but we saw significant performance degradation for both ingestion and query when doing compaction when it comes to finance data during trading hours.

Re: Apache Arrow is 10 years old

#72

Post by yencabulator » Fri, Feb 13, 2026, 4:35 PM UTC

amluto wrote: post #56

yencabulator wrote: post #53

Earlier quoted context omitted.

Frankly, RocksDB, SQLite or Postgres would be easy choices for that. (Fast) durable writes are actually a nasty problem with lots of little detail to get just right, or you end up with corrupted data on restart. For example, blocks may be written out of order so on a crash you may end up storing 12_4, and if you trust all content seen in the file, or even a footer in 4, you're screwed. Speaking as a Rustafarian, ther…

> Frankly, RocksDB, SQLite or Postgres would be easy choices for that. (Fast) durable writes are actually a nasty problem with lots of little detail to get just right, or you end up with corrupted data on restart. For example, blocks may be written out of order so on a crash you may end up storing 12_4, and if you trust all content seen in the file, or even a footer in 4, you're screwed. I have a WAL that works nicel…

And I have a system that launches Postgres in a container as part of a unit test (a little wrapper around https://crates.io/crates/pgtemp). It's much better than nothing, but the test using Postgres takes 0.5 seconds when the same business logic run against an in-memory implementation takes 0.005s.

Re: Apache Arrow is 10 years old

#73

Post by pitrou » Fri, Feb 13, 2026, 4:36 PM UTC

aynyc wrote: post #70

willtemperley wrote: post #65

Earlier quoted context omitted.

Feather (Arrow IPC) is zero copy and an order of magnitude simpler. Parquet has a lot of compatibility issues between readers and writers. Arrow is also directly usable as the application memory model. It’s pretty common to read Parquet into Arrow for transport.

When you say compatibility issues, you mean they are more problematic or less? It’s pretty common to read Parquet into Arrow for transport. I'm confused by this. Are you referring to Arrow Flight RPC? Or are you saying distributed analytic engine use arrow to transport parquet between queries?

Not the OP, but Parquet compatibility issues are usually due to the varying support of features across implementations. You have to take that into account when writing Parquet data (unless you go with the defaults which can be conservative and suboptimal).

Recently we have started documenting this to better inform choices: https://parquet.apache.org/docs/file-format/implementationst...

Re: Apache Arrow is 10 years old

#74

Post by kccqzy » Tue, Feb 17, 2026, 3:43 PM UTC

pitrou wrote: post #69

kccqzy wrote: post #25

Earlier quoted context omitted.

We use Apache Arrow at my company too. It is part of a migration from an old in-house format. When it works it’s good. But there are just way too many bugs in Arrow. For example: a basic arrow computation on strings segfaults because the result does not fit in Arrow’s string type, only the large string type. Instead of casting it or asking the user to cast it, it just segfaults. Another example: a different basic ope…

Hey, Arrow developer here. If you get a segfault with our codebase, then please report an issue on our GitHub issue tracker. (if you have already done so and it wasn't resolved, feel free to ping me on it)

Hey I just got back into work after the long weekend!

I thought a colleague of mine had filed an issue but I didn’t find it. I filed it myself just now: https://github.com/apache/arrow/issues/49310