FeatureBase uses Arrow for processing data stored in bitmap format: https://featurebase.com/
Demystifying Apache Arrow (2020)
31–40 of 49 posts
Re: Demystifying Apache Arrow (2020)
#32Author here. Since I wrote this, Arrow seems to be be more and more pervasive. As a data engineer, the adoption of Arrow (and parquet) as a data exchange format has so much value. It's amazing how much time me and colleagues have spent on data type issues that have arisen from the wide range of data tooling (R, Pandas, Excel etc. etc.). So much so that I try to stick to parquet, using SQL where possible to easily pre…
Since you know a bunch about this, I'm going to ask you a question that I was about to research: If I have a dataset in memory in Arrow, but I want to cache it to disk to read back in later, what is the most efficient way to do that at this moment in time? Is it to write to parquet and read the parquet back into memory, or is there a more efficient way to write the native Arrow format such that it can be read back in…
You're probably looking for the Arrow IPC format [1], which writes the data in close to the same format as the memory layout. On some platforms, reading this back is just an mmap and can be done with zero copying. Parquet, on the other hand, is a somewhat more complex format and there will be some amount of encoding and decoding on read/write. Flight is an RPC framework that essentially sends Arrow data around in IPC format.
Re: Demystifying Apache Arrow (2020)
#33Re: Demystifying Apache Arrow (2020)
#34Earlier quoted context omitted.
Since you know a bunch about this, I'm going to ask you a question that I was about to research: If I have a dataset in memory in Arrow, but I want to cache it to disk to read back in later, what is the most efficient way to do that at this moment in time? Is it to write to parquet and read the parquet back into memory, or is there a more efficient way to write the native Arrow format such that it can be read back in…
I'm not an expert in the nuts and bolts of Arrow, but I think you have two options: - Save to feather format. Feather format is essentially the same thing as the Arrow in-memory format. This is uncompressed and so if you have super fast IO, it'll read back to memory faster, or at least, with minimal CPU usage. - Save to compressed parquet format. Because you're often IO bound, not CPU bound, this may read back to mem…
Thanks! Exactly what I was looking for. I'll do some benchmarking of these two options for my workload.
Re: Demystifying Apache Arrow (2020)
#35> Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds If someone has a code example to this effect, I'd be greatful. I was once engaged in a salesy pitch by a cloud advocate that BigQuery (et al.) can "process a billion rows a second". I tried to create an SQLite example with a billion rows to show that this isn't impressive, but I gave up after some obstacles to gener…
[deleted]
Re: Demystifying Apache Arrow (2020)
#36> Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds If someone has a code example to this effect, I'd be greatful. I was once engaged in a salesy pitch by a cloud advocate that BigQuery (et al.) can "process a billion rows a second". I tried to create an SQLite example with a billion rows to show that this isn't impressive, but I gave up after some obstacles to gener…
You can try the examples or datafusion with flight. I have been able to process data with that setup in Rust under milliseconds that usually takes tens of seconds with a distributed query engine. I think Rust combined with Arrow, Flight, Parquet can be a game changer for analytics after a decade of Java with Hadoop & co.
completely agree with this. Rust and arrow will be part of the next set of toolsets for data engineering. Spark is great and I use it every day but it's big and cumbersome to use. There are use-cases today that are being addressed by datafusion, duckdb, (to a certain extent, pandas).. that will continue to evolve.. hopefully ballista can mature to a point where it's a real spark alternative for distributed computations. Spark isn't standing still of course and we're already seeing a lot of different drop in C++ SQL engines.. but moving entirely away from the JVM would be a watershed, IMO
Re: Demystifying Apache Arrow (2020)
#37Earlier quoted context omitted.
Clickhouse or DuckDB are databases I would look at that support this use case pretty much "out of the box" E.g. https://benchmark.clickhouse.com has some query times for a 100 million row dataset.
DuckDB is so simple to work with. It's only worth to look elsewhere with real big data, or where you really need a client-server setup. I hope it receives more love.
Duckdb is outrageously useful. Great on its own, but slots in perfectly reading and providing back arrow data frames, meaning you can seamlessly swap between tools when SQL for some parts and other tools better for others. Also very fast. I was able to throw away designs for multi machine setups as duckdb on its own was fast enough to not worry about anything else.
Re: Demystifying Apache Arrow (2020)
#38Can someone comment on the code quality of Arrow vs other Apache data engineering tools? I have been burned so many times by amateur hour software engineering failures from the Apache world, that it’s very hard for me to ever willingly adopt anything from that brand again. Just put it in gripped JSon or TSV and hey, if there’s a performance penalty, it’s better to pay a bit more for cloud compute than hate your job b…
Arrow the format is pretty good, there are occasional quirks (null bitmap has 1 = non-null etc) but no big deal.
From my experience Arrow the C++ implementation is pretty solid too, though I don't like it (taste). I just don't like their "force std::shared_ptr over Array, Table, Schema and basically everything" approach, why don't use an intrusive ref count if the object could only be hold by shared_ptr anyways? There are also a lot of const std::shared_ptr& arguments on not-obvious-when-it-takes-ownership functions. And immutable Array + ArrayBuilder (versus COW/switch between mutable uniquely owned and immutable shared in ClickHouse and friends), so if you have to fill the data out of order you are forced to buffer your data on your side.
Do note that the compute engine (e.g. Velox) may still need to implement their own (Arrow compatible) array types as there aren't many fancy encodings in Arrow the format.
Re: Demystifying Apache Arrow (2020)
#39Author here. Since I wrote this, Arrow seems to be be more and more pervasive. As a data engineer, the adoption of Arrow (and parquet) as a data exchange format has so much value. It's amazing how much time me and colleagues have spent on data type issues that have arisen from the wide range of data tooling (R, Pandas, Excel etc. etc.). So much so that I try to stick to parquet, using SQL where possible to easily pre…
Thanks for sharing your insights. Any comments on Feather vs Parquet? If we don't need to support tools that can only interact with Parquet, how will Feather pan out as a Parquet alternative (or Feather can't be such alternative at all)?
Re: Demystifying Apache Arrow (2020)
#40Here is another blog post that offers some perspective on the growth of Arrow over the intervening years and future directions: https://www.datawill.io/posts/apache-arrow-2022-reflection/