Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Demystifying Apache Arrow (2020)

robinlinacre.com

Post reply on HN
Print view View original

11–20 of 49 posts

Re: Demystifying Apache Arrow (2020)

#11

Post by mihevc »

mjburgess wrote: post #2

> Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds If someone has a code example to this effect, I'd be greatful. I was once engaged in a salesy pitch by a cloud advocate that BigQuery (et al.) can "process a billion rows a second". I tried to create an SQLite example with a billion rows to show that this isn't impressive, but I gave up after some obstacles to gener…

Here are some cookbook examples: https://arrow.apache.org/cookbook/py/data.html#group-a-table, https://arrow.apache.org/cookbook/. Datasets would probably be a good approach for the billions size, see: https://blog.djnavarro.net/posts/2022-11-30_unpacking-arrow-...

Re: Demystifying Apache Arrow (2020)

#12

Post by kajika91 »

I prefer looking at benchmarks : https://towardsdatascience.com/the-best-format-to-save-panda...

I have used Arrow and even made my humble contribution to the Go binding but I don't like pretending it is so much better than other solutions. It is not a silver bullet and probably the best pro is the "non-copy" goal to convert data into different frameworks' object. Depending of the use for the data columnar layout can be better but not always.

Re: Demystifying Apache Arrow (2020)

#13

Post by RobinL »

mjburgess wrote: post #2

> Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds If someone has a code example to this effect, I'd be greatful. I was once engaged in a salesy pitch by a cloud advocate that BigQuery (et al.) can "process a billion rows a second". I tried to create an SQLite example with a billion rows to show that this isn't impressive, but I gave up after some obstacles to gener…

An example using R code is here: https://arrow.apache.org/docs/r/articles/dataset.html

The speed comes from the raw speed of arrow, but also a 'trick'. If you apply a filter, this is pushed down to the raw parquet files so some don't need to be read at all due to the hive-style organisation

Another trick is that parquet files store some summary statistics in their metadata. This means, for example, that if you want to find the max of a column, only the metadata needs to be read, rather than the data itself.

I'm a Python user myself, but the code would be comparable on the Python side

Re: Demystifying Apache Arrow (2020)

#14

Post by tveita »

mjburgess wrote: post #2

> Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds If someone has a code example to this effect, I'd be greatful. I was once engaged in a salesy pitch by a cloud advocate that BigQuery (et al.) can "process a billion rows a second". I tried to create an SQLite example with a billion rows to show that this isn't impressive, but I gave up after some obstacles to gener…

Clickhouse or DuckDB are databases I would look at that support this use case pretty much "out of the box"

E.g. https://benchmark.clickhouse.com has some query times for a 100 million row dataset.

Re: Demystifying Apache Arrow (2020)

#16

Post by benjaminwootton »

Lyngbakr wrote: post #10
BeefWellington wrote: post #7

Earlier quoted context omitted.

Were you working off proper data warehouses, or just the transactional db? I ask because something a lot of people miss here is how much performance you can get from the T part of ETL. Denormalizing everything into big simple inflated tables makes things orders of magnitude faster. It matters quite a bit what your comparison is against.

We saw major improvements when we simply wrote full tables from a transactional database to parquet, but also, as you say, modelling the data appropriately produced significant improvements, too.

A column oriented database is probably the bigger performance increase. Parquet and a good data warehouse (something like Clickhouse, Druid or Snowflake) will both use metadata and efficient scans to power through aggregation queries.

Re: Demystifying Apache Arrow (2020)

#17

Post by RobinL »

Author here. Since I wrote this, Arrow seems to be be more and more pervasive. As a data engineer, the adoption of Arrow (and parquet) as a data exchange format has so much value. It's amazing how much time me and colleagues have spent on data type issues that have arisen from the wide range of data tooling (R, Pandas, Excel etc. etc.). So much so that I try to stick to parquet, using SQL where possible to easily preserve data types (pandas is a particularly bad offender for managing data types).

In doing so, I'm implicitly using Arrow - e.g. with Duckdb, AWS Athena and so on. The list of tools using Arrow is long! https://arrow.apache.org/powered_by/

Another interesting development since I wrote this is DuckDB.

DuckDB offers a compute engine with great performance against parquet files and other formats. Probably similar performance to Arrow. It's interesting they opted to write their own compute engine rather than use Arrow's - but I believe this is partly because Arrow was immature when they were starting out. I mention it because, as far as I know, there's not yet an easy SQL interface to Arrow from Python.

Nonetheless, DuckDB are still Arrow for some of its other features: https://duckdb.org/2021/12/03/duck-arrow.html

Arrow also has a SQL query engine: https://arrow.apache.org/blog/2019/02/04/datafusion-donation...

I might be wrong about this - but in my experience, it feels like there's more consensus around the Arrow format, as opposed to the compute side.

Going forward, I see parquet continuing on its path to becoming a de facto standard for storing and sharing bulk data. I'm particularly excited about new tools that allow you to process it in the browser. I've written more about this just yesterday: https://www.robinlinacre.com/parquet_api/, discussion: https://news.ycombinator.com/item?id=34310695.

Re: Demystifying Apache Arrow (2020)

#19

Post by spaniard89277 »

tveita wrote: post #14
mjburgess wrote: post #2

> Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds If someone has a code example to this effect, I'd be greatful. I was once engaged in a salesy pitch by a cloud advocate that BigQuery (et al.) can "process a billion rows a second". I tried to create an SQLite example with a billion rows to show that this isn't impressive, but I gave up after some obstacles to gener…

Clickhouse or DuckDB are databases I would look at that support this use case pretty much "out of the box" E.g. https://benchmark.clickhouse.com has some query times for a 100 million row dataset.

DuckDB is so simple to work with. It's only worth to look elsewhere with real big data, or where you really need a client-server setup.

I hope it receives more love.

Post reply on HN