Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow is 10 years old

arrow.apache.org

Post reply on HN
Print view View original

61–70 of 74 posts

Re: Apache Arrow is 10 years old

#61

Post by benrutter »

fud101 wrote: post #59
hodapp wrote: post #50

Earlier quoted context omitted.

The creator of Pandas even bashes it: https://wesmckinney.com/blog/apache-arrow-pandas-internals/

have these 'improvements' been backported to pandas now? i would expect it to close the gap over time.

Yes (mostly) is the answer. You can use arrow as a backend, and I think with v3 (recently released) it's the default.

The harder thing to overcome is that pandas has historically had a pretty "say yes to things" culture. That's probably a huge part of its success, but it means there are now about 5 ways to add a column to a dataframe.

Adding support for arrow is a really big achievement, but shrinking an oversized api is even more ambitious.

Re: Apache Arrow is 10 years old

#62

Post by benrutter »

aynyc wrote: post #30
tosh wrote: post #27

Earlier quoted context omitted.

parquet is optimized for storage and compresses well (=> smaller files) feather is optimized for fast reading

Given the cost of storage is getting cheaper, wouldn't most firms want to use feather for analytic performance? But everyone uses parquet.

There's definitely a "everyone uses it because everyone uses it" effect.

Feather might be a better fit for sime yse cases, but parquet has fantastic support and is still a pretty good choice for things that feather does.

Unless they're really focussed on eaking out every bit of read performance, people often opt for the well supported path instead.

Re: Apache Arrow is 10 years old

#63

Post by benrutter »

vivzkestrel wrote: post #54

stupid question: why hasnt apache arrow taken over to the point where we are not longer dealing with json?

I think a big reason (aside from intertia) is that arrow is designed for tables. Json sends a lot more than just that and can support whatever octagonal junitsu squid shaped data you want to fit into it.

Also, a good proportion of web apis are sending pretty small data sizes. On mass there might be an improvement if everything was more efficiently represented, but evaluating on a case by case basis, the data size often isn't the bottleneck.

Re: Apache Arrow is 10 years old

#64

Post by imtringued »

hodapp wrote: post #50
0xcafefood wrote: post #10

Earlier quoted context omitted.

Do people bash pandas? If so, it reminds me of Bjarne's quip that the two types of programming languages are the ones people complain about and the ones nobody uses.

The creator of Pandas even bashes it: https://wesmckinney.com/blog/apache-arrow-pandas-internals/

He missed talking about the poor extensibility of pandas. It's missing some pretty obvious primitives to implement your own operators without whipping out slow for loops and appending to lists manually.

Re: Apache Arrow is 10 years old

#65

Post by willtemperley »

aynyc wrote: post #21

What's the difference between feather and parquet in terms of usage? I get the design philosophy, but how would you use them differently?

Feather (Arrow IPC) is zero copy and an order of magnitude simpler. Parquet has a lot of compatibility issues between readers and writers.

Arrow is also directly usable as the application memory model. It’s pretty common to read Parquet into Arrow for transport.

Re: Apache Arrow is 10 years old

#66

Post by vb-8448 »

aynyc wrote: post #30
tosh wrote: post #27

Earlier quoted context omitted.

parquet is optimized for storage and compresses well (=> smaller files) feather is optimized for fast reading

Given the cost of storage is getting cheaper, wouldn't most firms want to use feather for analytic performance? But everyone uses parquet.

Storage is cheap but bandwidth no.

Re: Apache Arrow is 10 years old

#67

Post by rlh2 »

0xcafefood wrote: post #10
ayhanfuat wrote: post #8

Earlier quoted context omitted.

Indeed. feather was a library to exchange data between R and pandas dataframes. People tend to bash pandas but its creator (Wes McKinney) has changed the data ecosystem for the better with the learnings coming from pandas.

Do people bash pandas? If so, it reminds me of Bjarne's quip that the two types of programming languages are the ones people complain about and the ones nobody uses.

People also love to hate R but data.table is light years better than pandas in my view

Re: Apache Arrow is 10 years old

#68

Post by bootsmann »

jtbaker wrote: post #17
ayhanfuat wrote: post #8

Earlier quoted context omitted.

Indeed. feather was a library to exchange data between R and pandas dataframes. People tend to bash pandas but its creator (Wes McKinney) has changed the data ecosystem for the better with the learnings coming from pandas.

I know pandas has a lot of technical warts and shortcomings, but I'm grateful for how much it empowered me early in my data/software career, and the API still feels more ergonomic to me due to the years of usage - plus GeoPandas layering on top of it. Really, prefer DuckDB SQL these days for anything that needs to perform well, and feel like SQL is easier to grok than python code most of the time.

> Really, prefer DuckDB SQL these days for anything that needs to perform well, and feel like SQL is easier to grok than python code most of the time.

I switched to this as well and its mainly because explorations would need to be translated to SQL for production anyways. If I start with pandas I just need to do all the work twice.

Re: Apache Arrow is 10 years old

#69

Post by pitrou »

kccqzy wrote: post #25
mempko wrote: post #18

We use Apache Arrow at my company and it's fantastic. The performance is so good. We have terabytes of time-series financial data and use arrow to store it and process it.

We use Apache Arrow at my company too. It is part of a migration from an old in-house format. When it works it’s good. But there are just way too many bugs in Arrow. For example: a basic arrow computation on strings segfaults because the result does not fit in Arrow’s string type, only the large string type. Instead of casting it or asking the user to cast it, it just segfaults. Another example: a different basic ope…

Hey, Arrow developer here. If you get a segfault with our codebase, then please report an issue on our GitHub issue tracker.

(if you have already done so and it wasn't resolved, feel free to ping me on it)

Re: Apache Arrow is 10 years old

#70

Post by aynyc »

willtemperley wrote: post #65
aynyc wrote: post #21

What's the difference between feather and parquet in terms of usage? I get the design philosophy, but how would you use them differently?

Feather (Arrow IPC) is zero copy and an order of magnitude simpler. Parquet has a lot of compatibility issues between readers and writers. Arrow is also directly usable as the application memory model. It’s pretty common to read Parquet into Arrow for transport.

When you say compatibility issues, you mean they are more problematic or less?

It’s pretty common to read Parquet into Arrow for transport.

I'm confused by this. Are you referring to Arrow Flight RPC? Or are you saying distributed analytic engine use arrow to transport parquet between queries?

Post reply on HN