Demystifying Apache Arrow (2020)
robinlinacre.com
Demystifying Apache Arrow (2020)
1–10 of 49 posts
Re: Demystifying Apache Arrow (2020)
#2> Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds
If someone has a code example to this effect, I'd be greatful.
I was once engaged in a salesy pitch by a cloud advocate that BigQuery (et al.) can "process a billion rows a second".
I tried to create an SQLite example with a billion rows to show that this isn't impressive, but I gave up after some obstacles to generating the data.
It would be nice to have an example like this to show developers (, engineers) who have become accustomed to the extreme levels of CPU abuse today, to show that modern laptops really are supercomputers.
It should be obvious that a laptop can rival a data centre at 90% of ordinary tasks, that it isn't in my view, has a lot to do with the state of OS/Browser/App/etc. design & performance. Supercomputers, alas, dedicated to drawing pixels by way of a dozen layers of indirection.
Re: Demystifying Apache Arrow (2020)
#3> Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds If someone has a code example to this effect, I'd be greatful. I was once engaged in a salesy pitch by a cloud advocate that BigQuery (et al.) can "process a billion rows a second". I tried to create an SQLite example with a billion rows to show that this isn't impressive, but I gave up after some obstacles to gener…
You can see some of the benchmarks in DataFusion (part of the Arrow project and built with Arrow as the underlying in-memory format) https://github.com/apache/arrow-datafusion/blob/master/bench...
Disclaimer: I'm a committer on the Arrow project and contributor to DataFusion.
Re: Demystifying Apache Arrow (2020)
#4This has been a game changer for us. When our analysts run queries on parquets using Arrow they are orders of magnitude faster than equivalent SQL queries on databases.
Re: Demystifying Apache Arrow (2020)
#5> Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds If someone has a code example to this effect, I'd be greatful. I was once engaged in a salesy pitch by a cloud advocate that BigQuery (et al.) can "process a billion rows a second". I tried to create an SQLite example with a billion rows to show that this isn't impressive, but I gave up after some obstacles to gener…
Generally, operating on raw numbers in a columnar layout is very very fast, even if you just write it as a straightforward loop.
Re: Demystifying Apache Arrow (2020)
#6> Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds If someone has a code example to this effect, I'd be greatful. I was once engaged in a salesy pitch by a cloud advocate that BigQuery (et al.) can "process a billion rows a second". I tried to create an SQLite example with a billion rows to show that this isn't impressive, but I gave up after some obstacles to gener…
You can try the examples or datafusion with flight. I have been able to process data with that setup in Rust under milliseconds that usually takes tens of seconds with a distributed query engine. I think Rust combined with Arrow, Flight, Parquet can be a game changer for analytics after a decade of Java with Hadoop & co.
Re: Demystifying Apache Arrow (2020)
#7This has been a game changer for us. When our analysts run queries on parquets using Arrow they are orders of magnitude faster than equivalent SQL queries on databases.
Were you working off proper data warehouses, or just the transactional db?
I ask because something a lot of people miss here is how much performance you can get from the T part of ETL. Denormalizing everything into big simple inflated tables makes things orders of magnitude faster. It matters quite a bit what your comparison is against.
Re: Demystifying Apache Arrow (2020)
#8This has been a game changer for us. When our analysts run queries on parquets using Arrow they are orders of magnitude faster than equivalent SQL queries on databases.
Author here! I've actually just written a separate blog post on a similar topic!
https://www.robinlinacre.com/parquet_api/
Parquet seems to be on a path to become the de facto standard for storing and sharing bulk data - for good reason! (discussion: https://news.ycombinator.com/item?id=34310695)
Re: Demystifying Apache Arrow (2020)
#9This has been a game changer for us. When our analysts run queries on parquets using Arrow they are orders of magnitude faster than equivalent SQL queries on databases.
Author here! I've actually just written a separate blog post on a similar topic! https://www.robinlinacre.com/parquet_api/ Parquet seems to be on a path to become the de facto standard for storing and sharing bulk data - for good reason! (discussion: https://news.ycombinator.com/item?id=34310695 )
very cool post. thanks
Re: Demystifying Apache Arrow (2020)
#10This has been a game changer for us. When our analysts run queries on parquets using Arrow they are orders of magnitude faster than equivalent SQL queries on databases.
Were you working off proper data warehouses, or just the transactional db? I ask because something a lot of people miss here is how much performance you can get from the T part of ETL. Denormalizing everything into big simple inflated tables makes things orders of magnitude faster. It matters quite a bit what your comparison is against.
We saw major improvements when we simply wrote full tables from a transactional database to parquet, but also, as you say, modelling the data appropriately produced significant improvements, too.