Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow is 10 years old

arrow.apache.org

Post reply on HN
Print view View original

41–50 of 74 posts

Re: Apache Arrow is 10 years old

#41

Post by sudoshred »

pm90 wrote: post #16

Its nice to see useful, impactful interchange formats getting the attention and resources they need, and ecosystems converging around them. Optimizing serialization/deserialization might seem like a "trivial" task at first, but when moving petabytes of data they quickly become the bottlenecks. With common interchange formats, the benefits of these optimizations are shared across stacks. Love to see it.

Intuitively appreciating that these "boring fundamentals" are the default bottlenecks is a aign of senior+ swe capability.

Re: Apache Arrow is 10 years old

#42

Post by yencabulator »

aynyc wrote: post #38
yencabulator wrote: post #34

Earlier quoted context omitted.

Generally Parquet files are combined in an LSM style, compacting smaller files into larger ones. Parquet isn't really meant for the "journal" of level-0 append-one-record style storage, it's meant for the levels that follow.

So feather for journaling and parquet for long term processing?

You basically can't do row by row appends to any columnar format stored in a single file. You could kludge around it by allocating arenas inside the file but that's still a huge write amplification, instead of writing a row in a single block you'd have to write a block per column.

Re: Apache Arrow is 10 years old

#43

Post by gchamonlive »

HoldOnAMinute wrote: post #37

I read that entire page and I could not tell you what Apache Arrow is, or what it does.

The post celebrates Apache Arrow's 10 years anniversary, so it's assuming you already know what it is and what it does, which I think is fair. If you don't you can always refer to the docs.

Re: Apache Arrow is 10 years old

#44

Post by farsa »

aynyc wrote: post #30
tosh wrote: post #27

Earlier quoted context omitted.

parquet is optimized for storage and compresses well (=> smaller files) feather is optimized for fast reading

Given the cost of storage is getting cheaper, wouldn't most firms want to use feather for analytic performance? But everyone uses parquet.

Storage getting cheaper did not really reach the cloud providers and for self-hosting it has recently gotten even more expensive due to AI bs.

Re: Apache Arrow is 10 years old

#45

Post by amluto »

yencabulator wrote: post #42
aynyc wrote: post #38

Earlier quoted context omitted.

So feather for journaling and parquet for long term processing?

You basically can't do row by row appends to any columnar format stored in a single file. You could kludge around it by allocating arenas inside the file but that's still a huge write amplification, instead of writing a row in a single block you'd have to write a block per column.

You can do row by row appends to a Feather (Arrow IPC — the naming is confusing). It works fine. The main problem is that the per-append overhead is kind of silly — it costs over 300 bytes (IIRC) per append.

I wish there was an industry standard format, schema-compatible with Parquet, that was actually optimized for this use case.

Re: Apache Arrow is 10 years old

#46

Post by gregw2 »

yencabulator wrote: post #42
aynyc wrote: post #38

Earlier quoted context omitted.

So feather for journaling and parquet for long term processing?

You basically can't do row by row appends to any columnar format stored in a single file. You could kludge around it by allocating arenas inside the file but that's still a huge write amplification, instead of writing a row in a single block you'd have to write a block per column.

Agreed.

There is room still for an open source HTAP storage format to be designed and built. :-)

Re: Apache Arrow is 10 years old

#47

Post by yencabulator »

amluto wrote: post #45
yencabulator wrote: post #42

Earlier quoted context omitted.

You basically can't do row by row appends to any columnar format stored in a single file. You could kludge around it by allocating arenas inside the file but that's still a huge write amplification, instead of writing a row in a single block you'd have to write a block per column.

You can do row by row appends to a Feather (Arrow IPC — the naming is confusing). It works fine. The main problem is that the per-append overhead is kind of silly — it costs over 300 bytes (IIRC) per append . I wish there was an industry standard format, schema-compatible with Parquet, that was actually optimized for this use case.

Creating a new record batch for a single row is also a huge kludge leading to lot of write amplification. At that point, you're better off storing rows than pretending it's columnar.

I actually wrote a row storage format reusing Arrow data types (not Feather), just laying them out row-wise not columnar. Validity bits of the different columns collected into a shared per-row bitmap, fixed offsets within a record allow extracting any field in a zerocopy fashion. I store those in RocksDB, for now.

https://git.kantodb.com/kantodb/kantodb/src/branch/main/crat...

https://git.kantodb.com/kantodb/kantodb/src/branch/main/crat...

https://git.kantodb.com/kantodb/kantodb/src/branch/main/crat...

Re: Apache Arrow is 10 years old

#48

Post by sirfz »

jtbaker wrote: post #17
ayhanfuat wrote: post #8

Earlier quoted context omitted.

Indeed. feather was a library to exchange data between R and pandas dataframes. People tend to bash pandas but its creator (Wes McKinney) has changed the data ecosystem for the better with the learnings coming from pandas.

I know pandas has a lot of technical warts and shortcomings, but I'm grateful for how much it empowered me early in my data/software career, and the API still feels more ergonomic to me due to the years of usage - plus GeoPandas layering on top of it. Really, prefer DuckDB SQL these days for anything that needs to perform well, and feel like SQL is easier to grok than python code most of the time.

chdb's new DataStore API looks really neat (drop in pandas replacement) and exactly how I envisioned a faster pandas could be without sacrificing its ergonomics

Re: Apache Arrow is 10 years old

#49

Post by aynyc »

dionian wrote: post #40
aynyc wrote: post #26

Earlier quoted context omitted.

I read that. But afaik, feather format is stable now. Hence my confusion. I use parquet at work a lot, where we store a lot of time series financial data. We like it. Creating the Parquet data is a pain since it's not append-able.

Have you considered something like iceberg tables?

Yes, but parquet hates small files.

Re: Apache Arrow is 10 years old

#50

Post by hodapp »

0xcafefood wrote: post #10
ayhanfuat wrote: post #8

Earlier quoted context omitted.

Indeed. feather was a library to exchange data between R and pandas dataframes. People tend to bash pandas but its creator (Wes McKinney) has changed the data ecosystem for the better with the learnings coming from pandas.

Do people bash pandas? If so, it reminds me of Bjarne's quip that the two types of programming languages are the ones people complain about and the ones nobody uses.

The creator of Pandas even bashes it: https://wesmckinney.com/blog/apache-arrow-pandas-internals/
Post reply on HN